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ABSTRACT 

Uncertainty  is  inherent  in  all  real  world  settings  and  it  creates  ambiguities  that  make  decision 
making  complex  and  difficult.  For  decades,  uncertainty  visualisation  has  been  a  prominent 
topic  in  the  research  of  military  decision  making.  Even  if  data  is  free  from  uncertainty,  errors 
can  occur  in  the  process  of  turning  the  data  into  the  'picture'.  Ignoring  the  fact  of  information 
uncertainty  could  lead  to  severe  consequences  in  the  military  domain.  This  report  presents  an 
overview  of  the  theoretical  concepts  and  definitions  of  uncertainty,  and  the  uncertainty 
visualisation  techniques  that  have  been  investigated  to  date.  It  discusses  the  literature  on  the 
impact  of  uncertainty  visualisation  on  decision  making,  and  introduces  recommended 
guidelines  and  systematic  strategies  for  uncertainty  visualisation.  Finally,  it  discusses  how 
this  work  can  be  applied  to  support  situation  awareness  and  decision  making  in  the 
Australian  Defence  Force's  Joint  Operations  Command. 
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Visualising  Uncertainty  for  Decision  Support 
Executive  Summary 

Military  commanders  are  typically  required  to  make  critical  decisions  and  develop 
plans,  using  large  and  complex  collections  of  data,  in  a  limited  time  frame.  They  must 
do  this  without  precise  knowledge  about  the  operating  environment  or  the  intent, 
capabilities  or  location  of  the  adversary.  It  is  important  that  commanders  understand 
the  associated  uncertainties  so  that  they  can  understand  and  mitigate  the  operational 
risks  involved  in  this  inherently  risky  enterprise.  Therefore  it  is  important  for  decision 
support  tools  to  make  users  aware  of  the  uncertainties  in  the  information  being 
displayed  in  an  appropriate  and  timely  manner,  while  avoiding  information  overload. 
However,  most  visualisation  techniques  have  been  designed  around  the  assumption 
that  the  data  being  visualised  is  free  from  uncertainty.  This  report  reviews  the 
concepts,  techniques,  and  effectiveness  of  visualisation  approaches  for  uncertainty 
presented  in  the  literature,  and  looks  at  how  these  could  be  applied  to  enhance 
situation  awareness  in  the  Joint  Operations  Command  (JOC). 

Uncertainty  can  come  in  many  forms,  with  11  different  types  of  uncertainty  discussed 
in  the  literature: 

•  Accuracy  -  the  difference  between  observation  and  reality 

•  Precision  -  the  quality  of  the  estimate  or  measurement 

•  Completeness  -  the  extent  to  which  information  is  comprehensive 

•  Consistency  -  the  extent  to  which  information  elements  agree 

•  Lineage  -  the  pathway  through  which  information  has  been  passed 

•  Currency  -  the  time  span  from  occurrence  to  information  presentation 

•  Credibility  -  the  reliability  of  the  information  source 

•  Subjectivity  -  the  extent  to  which  the  observer  influences  the  observation 

•  Interrelatedness  -  the  dependence  on  other  information 

•  Experimental  -  the  width  of  a  random  distribution  of  observations 

•  Geometric  -  the  region  within  which  a  spatial  observation  lies. 

Each  of  these  different  types  of  uncertainty  applies  to  different  types  of  information, 
and  can  be  quantified,  and  thus  represented,  in  different  ways.  These  uncertainties  can 
be  introduced  at  any  stage  during  an  information  processing  pipeline: 

•  Acquisition  -  introduced  by  the  measurement  or  sampling  processes 

•  Transformation  -  introduced  by  processing  algorithms  or  fusion  processes 

•  Visualisation  -  introduced  by  visualisation  artefacts  or  filters. 
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Indeed,  the  act  of  trying  to  account  for,  quantify,  and  visualise  uncertainties  could 
potentially  introduce  more  uncertainties  into  a  system,  and  so  it  is  important  to 
understand  the  nature  of  any  uncertainties  and  their  impact  on  the  quality  of  decision 
making.  It  is  particularly  important  to  ensure  that  attempting  to  represent  uncertainty 
does  not  introduce  artefacts  that  obscure,  clutter,  or  interfere  with  the  information  to  be 
displayed.  Visualisation  approaches  that  have  been  used  for  representing  uncertainty 
fall  into  two  general  categories: 

•  intrinsic  representation  techniques  that  integrate  uncertainty  by  varying  the 
appearance  of  the  data  (e.g.  shape,  texture,  brightness,  opacity  and  hue) 

•  extrinsic  representation  techniques  that  add  geometry  to  describe  the 
uncertainty  (e.g.  arrows,  error  bars  and  charts). 

The  choice  of  visualisation  approach  depends  on  the  nature  of  the  uncertainty  and  the 
application  context.  For  geospatial  contexts,  five  intuitive  categories  of  uncertainty 
representation  have  been  suggested: 

•  modification  of  graphical  attributes,  such  as  colour,  texture,  blurring,  and 
opacity 

•  addition  of  artefacts,  such  as  glyphs,  contours,  and  iso-surfaces 

•  animation  of  graphical  attributes  to  illustrate  the  expected  variability 

•  non-visual  techniques,  such  as  acoustic  and  haptic  feedback  coupled  with  a 
visual  display 

•  user  interaction,  such  as  information  pop-ups  on  mouse  hover  over  a  data 
feature. 

Much  of  the  work  covered  in  the  literature  has  focussed  on  the  technical  feasibility  and 
implementation  of  these  approaches,  with  little  analysis  of  their  perceptual  or  cognitive 
value,  and  little  systematic  evaluation  done  on  the  general  effectiveness  of  these 
approaches.  Thus,  studies  need  to  be  tailored  to  particular  user  contexts.  There  have 
been  empirical  studies  carried  out  on  the  effect  of  uncertainty  on  decision  making, 
which  found  marked  differences  in  performance  between  experienced  and 
inexperienced  users.  The  decision  times  for  experienced  users  were  not  affected  by 
uncertainty,  while  those  of  inexperienced  users  were  significantly  increased. 
Interestingly,  visualisation  of  uncertainty  has  been  found  to  improve  decision  making 
performance  for  relatively  easy  tasks,  but  not  when  dealing  with  more  difficult  tasks 
when,  perhaps,  other  considerations  dominate.  How  user  experience  maps  to 
subjective  task  difficulty  could  be  an  interesting  consideration  that  makes  the  selection 
of  participants  for  empirical  studies  particularly  important  when  studying  the 
effectiveness  of  uncertainty  representation  techniques.  Clearly,  participants  in 
empirical  studies  need  to  be  representative  of  the  skills  and  experience  of  the  target 
user  group. 

Several  empirical  studies  have  also  looked  at  the  effects  of  different  uncertainty 
visualisation  approaches  on  decision  making  in  different  contexts.  Dynamic 
uncertainty  representation  techniques  (e.g.  animation)  were  generally  found  to  be  less 
effective  than  static  techniques  (e.g.  glyphs)  in  decision  making  tasks.  Somewhat 
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surprisingly,  the  addition  of  textual  annotations  of  confidence  to  a  glyph  representing 
the  degree  of  uncertainty  was  found  to  give  poorer  performance  than  the  glyph  alone 
in  a  target  identification  task,  perhaps  indicating  information  overload.  In  other 
studies,  different  uncertainty  representations  were  also  found  to  suit  different  user 
requirements  in  the  same  application,  suggesting  that  some  degree  of  tailoring  may  be 
required  to  meet  multiple  user  roles.  This  should  be  treated  with  caution  however,  as 
studies  have  also  shown  that  user  satisfaction  with  information  products  does  not 
necessarily  coincide  with  improved  situational  awareness  in  general,  and  user 
preferences  for  uncertainty  representation  techniques  do  not  necessarily  coincide  with 
improved  decision  making  performance  in  particular.  System  metrics  and  governance 
frameworks  may  be  needed  to  help  manage  this  complexity. 

Other  approaches  to  uncertainty  visualisation  could  use  the  method  of  representation 
itself  to  attribute  confidence  in  the  information  presented.  For  example,  users  have 
greater  confidence  in  information  presented  textually  than  the  same  information 
presented  by  a  virtual  human  avatar.  Thus,  in  this  approach  information  with  low 
uncertainty  could  be  presented  to  the  user  as  text,  while  information  with  high 
uncertainty  could  be  presented  to  the  user  by  an  avatar.  Further  studies  could  also 
explore  how  users  attribute  confidence  to  information  presented  using  other 
visualisation  modalities. 

There  are  limited  guidelines  available  for  the  development  of  applications  utilising  the 
various  approaches  considered  when  representing  multiple  types  of  uncertainty  in  a 
decision  making  context.  More  study  is  needed  in  how  to  depict  multiple  forms  of 
uncertainty  in  the  same  display,  and  how  adding  multiple  types  of  uncertainties  to 
visual  displays  affects  the  users'  understanding.  General  guidelines  for  visualisation 
can  be  applied  to  help  formulate  an  uncertainty  visualisation  approach  suitable  for  a 
particular  context,  once  the  relevant  components  of  uncertainty,  their  relationships  to 
the  data,  and  the  desired  decision  making  outcomes  are  understood.  User  centred 
design,  iterative  approaches  to  development,  and  methods  for  assessing  relevant 
performance  metrics,  are  considered  crucial  given  the  sensitivity  of  decision  making 
outcomes  to  context,  user  experience,  and  user  roles. 

The  situational  awareness  requirements  of  the  Australian  Defence  Force's  Joint 
Operations  Command  (JOC)  range  across  the  spectrum  of  ADF  operations  to  achieve 
national  strategic  objectives.  The  users  in  JOC  need  to  access,  integrate  and  visualise  a 
diversity  of  information  across  the  Defence  Enterprise  including  military  text  messages, 
unstructured  documents,  military  databases  and  open-source  content  from  civilian 
producers.  Furthermore,  this  needs  to  be  tailored  to  support  a  variety  of  dynamic 
operational  needs,  because  the  users  are  unlikely  to  be  subject  matter  experts  in  each  of 
the  data  sets  being  integrated.  It  is  crucial  that  they  understand  the  uncertainties 
associated  with  the  information  being  displayed  so  that  they  can  mitigate  operational 
risks. 

In  this  context,  the  sources  of  uncertainty  include: 

•  Incompleteness  due  to  parsing  errors  from  poorly  formatted  structured  content 

•  Geometric  uncertainties  associated  with  geospatial  content 
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•  Ambiguities  and  incorrect  entity  associations  from  unstructured  content 

•  Incomplete  metadata 

•  Inconsistent  or  incompatible  schema  and/or  standards  associated  with 
structured  content 

•  Inconsistent  information  from  multiple  sources 

•  Interrelatedness  of  content  and/  or  corrections 

•  Currency  of  content 

•  Lineage  and/  or  credibility  of  content 

•  Incompleteness  where  data  aggregation  would  raise  the  classification  above  the 
system's  accreditation 

•  Incompleteness  due  to  the  pagination  of  search  results,  map  scale  and/or 
viewpoint  settings  in  geospatial  displays 

•  Incompleteness  due  to  object  clustering  used  to  reduce  visual  clutter 

•  Interference  between  overlapping  visualisation  layers  and/or  overlapping 
symbology. 

Some  of  the  approaches  discussed  in  the  literature  that  could  be  used  to  visualise  these 
uncertainties  include: 

•  Combining  representation  techniques  such  as  opacity,  blurring,  and  degraded 
icons  to  visualise  compounded  uncertainty  (e.g.  geospatial  uncertainty, 
information  currency,  and  status  ambiguity  in  incident  reports  or  emergency 
events) 

•  Highlighting  inconsistencies  in  the  information  presented  through  colour 
coding 

•  Highlighting  the  credibility/ lineage  of  content  through  colour  coding 

•  Providing  a  visual  indicator  of  missing/incomplete  information,  with 
elaboration  provided  through  user  interaction  (e.g.  mouse  hover/ click) 

•  Providing  multimodal  content  through  audible  or  haptic  feedback  while 
interacting  with  the  visual  display 

•  Using  different  visualisation  modalities  to  confer  different  levels  of  confidence 
in  the  information  presented  (e.g.  use  of  an  avatar  to  confer  lower  levels  of 
confidence  in  uncertain  information). 

•  Allowing  users  to  choose  appropriate  uncertainty  visualisation  techniques  that 
suit  their  current  task  and  experience,  based  on  user  and  role  models. 

Empirical  studies  and  further  conceptual  refinement  are  needed  to  target  the  particular 
requirements  of  JOC  users.  Significant  barriers  to  these  studies  will  be  the  availability 
of  experienced  users  for  empirical  evaluation,  and  the  dynamic  nature  of  their 
visualisation  requirements  across  a  broad  range  of  information  types  and  associated 
uncertainties.  The  dynamic  nature  of  these  requirements  means  that  many  of  these 
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users  may  be  unfamiliar  with  a  particular  task  or  display,  which  may  make  it  easier  to 
find  participants  representative  of  this  target  group.  It  also  indicates  that  visualisation 
of  the  associated  uncertainties  has  the  potential  to  provide  significant  benefits  to 
decision  making  performance  in  this  context. 

Finally,  how  decision  making  with  uncertainty  translates  to  operational  risk  is  an  area 
that  requires  further  study.  In  the  studies  presented  in  the  literature  the  performance  of 
the  decision  maker  was  evaluated  using  metrics  such  as  response  time  and  the  user's 
confidence  in  the  outcome.  Whether  the  representation  of  uncertainty  translates  to  a 
better  appreciation  of  operational  risk,  and  more  effective  mitigation  strategies,  has  not 
been  considered  in  this  work.  Studies  that  model  and  evaluate  operational  risk,  and/  or 
include  operational  red-teaming,  could  help  address  this  question. 
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1.  Introduction 


1.1  Motivation 

Visualisation  is  the  communication  of  data  in  a  visible  form  that  brings  out  relationships 
and  features  in  data.  The  main  goal  of  data  visualisation  is  to  communicate  information 
clearly  and  effectively  through  graphical  means  (Friedman,  2008).  Effective  visualisation 
makes  massive  and  complex  data  understandable  and  usable  in  an  intuitive  way  to  help 
users  analyse  and  reason  about  it. 

Most  scientific  data  sets  are  not  considered  complete  without  an  indication  of  some 
measure  of  uncertainty  such  as  error,  accuracy  or  level  of  confidence.  However,  traditional 
visualisation  approaches  have  usually  been  developed  based  on  the  assumption  that  all 
data  are  exact  and,  thus,  the  visual  representation  of  uncertainty  has  been  overlooked 
(Lapinski,  2009).  Uncertainty  could  be  depicted  in  a  variety  of  formats  ranging  from 
simple  text  to  dynamic  graphical  representations.  Increasingly,  understanding  how  to 
effectively  display  uncertainty  information  is  becoming  an  important  topic  across  many 
domains  such  as  cartography,  geographic  visualisation  and  scientific  visualisation. 

According  to  Finger  and  Bisantz  (2002)  the  question  of  how  to  represent  uncertainty  is 
important  from  two  perspectives.  Firstly,  there  is  the  theoretical  need  to  determine  the 
manner  in  which  different  representations  or  display  formats  may  impact  users' 
understanding  of  the  uncertainty.  This  subsequently  affects  the  decisions  and  actions 
made  based  on  these  different  representations.  Secondly,  it  is  necessary  to  determine  how 
best  to  display  this  information  to  users,  particularly  when  there  is  uncertainty  associated 
with  a  large  number  of  objects  or  data  points. 

Indications  of  uncertainty  in  visualisation  are  crucial  to  understanding  the  "reliability"  of 
information,  and  consequently  affect  decision  making  (Deitrick,  2007).  Olston  and 
Mackinlay  (2002)  argued  that  information  visualisation  tools  should  make  users  aware  of 
the  presence,  nature,  and  degree  of  uncertainty  in  the  data,  otherwise  users  may  draw 
inaccurate  conclusions,  potentially  leading  to  costly  mistakes.  Indeed,  command  and 
control  (C2)  issues  in  regards  to  uncertainty  visualisation  have  long  been  regarded  as  a 
difficult  topic  since  the  commander  has  to  make  decisions  in  a  limited  time  frame  with 
information  that  comes  from  varied  sources,  in  many  formats,  with  a  continuum  of 
validity  (Summers  et  al.,  2005). 

As  decision  makers  become  more  reliant  on  data  visualisation  capabilities  to  support  their 
decision  making,  the  importance  of  visualising  uncertainty  in  the  data  grows.  However, 
the  actual  task  of  visualising  this  uncertainty  becomes  problematic  as  it  is  not  easy  to 
include  additional  uncertainty  information  into  an  existing  visualisation  without 
introducing  visual  clutter  and  impairing  comprehensibility  (Riveiro,  2007).  Uncertainty 
visualisation  increases  the  dimensionality  of  the  data  (e.g.  multivariate  and  statistical 
visualisation  techniques  (Pang,  2001)),  and  the  richness  needed  in  the  display  to  avoid 
obscuration  of  information. 
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In  this  report  we  review  some  of  the  approaches  suggested  for  visualising  uncertainty,  and 
explore  how  they  may  be  applied  to  decision  support  tools.  We  begin  by  summarising  the 
different  types  and  sources  of  uncertainty  identified  in  the  literature.  We  then  examine 
different  methodologies  that  have  been  suggested  for  the  visualising  various  types  of 
uncertainty  relevant  to  situation  awareness.  Next  we  examine  specific  empirical  studies  on 
the  effect  of  uncertainty  on  decision  making,  and  the  effectiveness  of  some  of  the 
approaches  used  for  uncertainty  visualisation  in  this  context.  We  then  suggest  some 
guidelines  and  strategies  for  uncertainty  visualisation  design,  and  how  these  techniques 
can  be  utilised  for  situation  awareness  in  the  Australian  Defence  Force's  Joint  Operations 
Command. 
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2.  Information  Uncertainty 

2.1  Types  of  uncertainty 

Good  practice  requires  statements  of  accuracy  by  which  the  reliability  of  results  can  be 
understood  and  communicated.  In  MacEachren  et  al.  (2005),  when  inaccuracy  is  known 
objectively  it  can  be  expressed  as  "error"  otherwise  the  term  "uncertainty"  applies.  Hunter 
and  Goodchild  (1993)  define  uncertainty  as  the  degree  to  which  the  lack  of  knowledge 
about  the  amount  of  error  is  responsible  for  hesitancy  in  accepting  results  and 
observations. 

In  Pang  et  al.  (1997),  the  authors  considered  three  types  of  uncertainty  in  their  discussion: 

•  statistical  -  either  given  by  the  estimated  mean  and  standard  deviation  which  can 
be  used  to  calculate  a  confidence  interval  or  an  actual  distribution  of  the  data 

•  error  -  a  difference  between  a  known  correct  datum  and  an  estimate 

•  range  -  an  interval  in  which  the  data  exist,  but  which  cannot  be  quantified  into 
either  the  statistical  or  error  definitions. 

In  general,  uncertainty  is  understood  as  a  composition  of  different  concepts.  It  has  many 
interpretations  across  various  domain  and  application  contexts  (Griethe,  2005).  Thomson 
et  al.  (2005)  have  constructed  a  typology  for  uncertainty  based  on  Gershon's  taxonomy 
(Gershon,  1998)  and  Pang  et  al.'s  classification  (Pang  et  al.,  1997)  (see  Table  1). 


Table  1  Term  and  general  definitions  for  uncertainty  categories  (modified  from  (Thomson  et  al., 
2005)) 


Category 

Definition 

Attribute 

Examples 

Location 

Examples 

Time 

Examples 

Accuracy /error 

Difference  between  observation 
and  reality,  usually  estimated 
based  on  knowledge  of  the 
measurement  /  estimation  device 
and  of  phenomena  in  the  work. 

counts, 

magnitudes 

coords., 

buildings 

+/- 1  day 

Precision 

Exactness  of  measurement 
/  estimate,  derived  from  parameters 
of  the  measurement,  estimation 
device,  and/or  procedure. 

nearest  1000 

1  degree 

once  per  day 

Completeness 

Extent  to  which  information  is 
comprehensive. 

75% 

reporting 

20%  cloud 
cover1 

5  samples  in 

100 

Consistency 

Extent  to  which  information 
components  agree.  This  is  a  more 
general  definition  than  that  found 
in  formal  standards  for  spatial  data 

multiple 

classifiers 

'from  a 
place'  vs 
'for  a  place' 

5  say  M; 

2  say  T 

1  Does  the  20%  measure  in  this  case  tell  you  that  there  is  a  20%  chance  of  100%  cloud  cover  at  any 
location,  or  that  at  any  location  there  is  a  100%  chance  of  20%  cloud  cover? 
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Lineage2 

Conduit  through  which 
information  has  passed.  This  is  a 
complex  category  that  has  at  least 
the  following  subcomponents: 
number  of  individuals, 
organizations,  processes  through 
which  information  moves; 
specification  of  which  individuals, 
organizations,  or  processes 

transformati 

ons 

#  of  input 
sources 

#  of  steps 

Currency 

Currency  depends  on  the  time  span 
from  occurrence  through 
information  collection  /  processing 
to  use,  and  the  context.  E.g.,  year- 
old  data  about  vehicles  parked  in  a 
factory  loading  bay  is  less  certain  to 
be  current  than  year-old  data  about 
location  of  the  factory 

census  data 

date  of  map 
production 

5  hours  ago 

Credibility 

Combination  of  factors  such  as 
reliability  of  information  source. 
Certainty  may  be  based  on  past 
experience,  e.g.,  the  analyst  is 
correct  85  percent  of  the  time,  or  on 
categorization  of  the  source,  e.g., 

U.S.  analyst  versus  a  non-U. S. 
informant;  motivation,  experience, 
or  other  factors. 

U.S.  analyst; 
informant 

knowledge 
of  place 

reliability  of 
model 

Subjectivity 

The  extent  to  which  human 
interpretation  or  judgment  is 
involved  in  information 
construction.  This  component  of 
uncertainty  is,  of  course,  difficult  to 
assess  —  and  that  assessment  will 
have  some  level  of  subjectivity. 

fact  /  guess 

local  / 
outsider 

experienced/ 

inexperienced 

Interrelatedness 

Source  independence  from  other 
information.  This  is  a  common 
standard  used  in  the  news  media  to 
assess  certainty  that  a  story  is 
authentic 

same  author 

source 

proximity 

time  proximity 

Potter  (2010)  suggested  two  additional  types  of  uncertainty,  depending  on  how  they  arise 
in  the  data  set: 

•  Experimental  uncertainty 

In  mathematics,  uncertainty  is  closely  related  to  probability  theory  which  describes 
the  occurrence  of  some  events  as  random.  Foody  and  Atkinson  (2002)  simply 
described  uncertainty  as  a  'quantitative  statement  about  the  probability  of  error'. 
Taylor  and  Kuyatt  (1993)  proposed  guidelines  for  expressing  the  uncertainty  of 
measurement  results  and  defined  experimental  uncertainty  as  the  standard 
deviation  of  a  collection  of  measured  results.  This  type  of  uncertainty  is  derived 


2  This  is  also  known  as  'provenance'. 
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from  running  an  experiment  numerous  times,  or  performing  a  non-deterministic 
simulation  in  which  the  outcome  varies  after  each  run  (Potter  et  al.,  2012). 

•  Geometric  uncertainty 

Geometric  uncertainty  arises  when  the  spatial  position  of  some  or  all  of  the  data  set 
is  in  question.  This  type  of  uncertainty  may  describe  the  amount  of  possible 
variation  between  a  data  point  and  the  true  location,  or  may  describe  a  boundary 
region  within  which  the  data  point  will  be  positioned.  For  example.  Figure  1 
illustrates  that  the  yellow  spheres  represent  the  spatial  uncertainty  of  the  points  on 
the  surface.  This  technique  could  be  useful  to  understand  and  analyse  data  more 
intuitively  because  the  user  can  understand  not  only  where  the  location  of  the 
surface  is,  but  also  the  relative  quantity  of  uncertainty  that  exists  at  each  point 
(Potter,  2010). 


Figure  1:  Tire  spheres  are  centred  on  the  data  points  and  their  size  shows  the  size  of  the  uncertainty. 
(Potter,  2010) 

2.2  Sources  of  uncertainty 

There  are  many  possible  sources  of  uncertainty  that  arise  from  the  quality  and  consistency 
of  data  sources.  Reveiro  (2007)  suggested  that  several  different  errors  and  uncertainties  in 
a  generic  information  system  can  be  introduced  at  different  stages  of  a  visualisation 
pipeline.  This  pipeline  is  broadly  divided  into  three  stages:  data  acquisition,  data 
transformation  and  data  visualisation  as  shown  in  Figure  2. 
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Figure  2:  Visualisation  pipeline  representing  the  introduction  of  data  uncertainty  from  the  model 
and  measurements  (acquisition),  derived  uncertainty  that  results  from  the 
transformation  of  the  data,  and  visualisation  uncertainty  that  results  from  the 
visualisation  process  itself  (Riveiro,  2007) 

Uncertainty  can  be  introduced  at  any  stage  of  this  pipeline  from  acquisition  to 
visualisation  as  follows: 

•  Data  acquisition  stage 

Uncertainties  occur  from  the  models  used  and  the  measurements  made  at  the  time 
of  acquisition.  For  example,  uncertainty  could  arise  when:  the  sensors  have  limited 
resolution;  their  readings  contain  noise;  their  measurements  may  be  uncertain  due 
to  miscalibration;  and  sampling  is  sparse  in  time  and  space. 

•  Data  transformation  stage 

In  most  cases,  raw  data  is  not  directly  rendered  without  further  transformation. 
Numerical  calculations,  for  example,  can  introduce  errors  from  the  limited 
precision  of  numerical  integration  algorithms,  or  distortions  can  be  introduced  that 
amplify  some  features  and  de-emphasise  others.  The  process  of  converting  raw 
data  into  a  suitable  form  may  involve:  normalising,  non-linear  scaling,  averaging; 
interpolating;  resampling;  and  quantising.  Note  that  these  transformation 
operations  may  occur  as  early  as  the  data  acquisition  stage  or  as  late  as  the 
visualisation  stage. 

•  Data  visualisation  stage 

Visualisation  itself  can  be  a  source  of  uncertainty.  In  this  case,  uncertainty  is 
generally  associated  with  the  rendering  models  and  algorithms  used  to  generate 
the  visualisation.  For  example,  rendered  3D  scenes  will  appear  different  when 
different  radiosity  algorithms  are  used  to  determine  the  global  illumination  (Arvo 
et  al.,  1994).  Similarly,  different  approaches  to  direct  volume  rendering  of  3D  data 
sets  could  yield  different  results  (Moller  et  al.,  1996).  One  good  example  is 
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illustrated  in  Figure  3  (Lundstrom  et  al.,  2007).  An  artery  is  shown  in  (a)  which 
appears  to  be  blocked  and  in  need  of  surgical  intervention.  However  in  (b),  by 
simply  changing  visualisation  parameters,  the  artery  now  appears  to  be  healthy. 


(a)  (b) 

Figure  3:  Uncertainty  from  visualisation  is  illustrated  in  this  example  showing  a  possibly  blocked 
artery  (a),  or  unblocked  artery  (b)  using  different  visualisation  parameters  (Lundstrom 
et  al,  2007). 

2.3  Discussion 

There  have  been  many  other  efforts  to  classify  the  sources  and  types  of  uncertainty.  New 
types  of  uncertainties  are  encountered  by  analysts  as  new  ways  of  processing  data  are 
developed.  There  are  also  additional  concepts  of  uncertainty  that  have  been  suggested, 
such  as:  fuzziness;  plausibility;  belief;  or  necessity  (Yao  et  al.,  1995,  Griethe  and  Schumann, 
2006).  As  uncertainty  becomes  more  abstract,  it  becomes  more  difficult  to  represent,  but  it 
may  also  have  less  relevance  to  data  visualisation.  In  the  following  chapters  we  will  focus 
primarily  on  those  types  of  uncertainty  relevant  to  data  visualisation  and  decision  making. 

The  nature  of  the  uncertainty  associated  with  the  data  discussed  in  the  previous  sections 
determines  how  it  needs  to  be  represented.  In  the  next  chapter  we  will  discuss  some  of  the 
visualisation  approaches  used  to  help  analysts  and  decision  makers  manage  and 
understand  imperfect  information. 
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3.  Uncertainty  Visualisation  Techniques 

3.1  Background 

There  have  been  many  attempts  to  categorise  the  approaches  used  for  uncertainty 
visualisation.  Gershon  (1998)  proposed  two  general  categories  of  uncertainty  visualisation 
techniques:  intrinsic  and  extrinsic.  Intrinsic  representation  techniques  integrate 
uncertainty  in  the  display  by  varying  an  object's  appearance,  for  example  by  varying 
visual  variables  such  as  texture,  brightness,  hue  or  shape.  Extrinsic  techniques  add 
geometric  objects,  including  arrows,  bars,  and  complex  objects  (such  as  pie  charts)  to 
represent  uncertainty. 

Pang  et  al.  (1997)  simply  categorised  uncertainty  visualisation  methods  into  seven 
categories: 

1.  Add  glyphs  (icons)  to  the  visualisation  to  represent  uncertainty  (e.g.  error  bars) 

2.  Add  geometry  to  show  the  possible  variation  due  to  uncertainty  (e.g.  maximum 
and  minimum  curves  in  a  data  plot) 

3.  Modify  geometry  to  indicate  the  degree  of  uncertainty  in  the  geometry  (e.g.  using 
dotted  lines  for  a  curve) 

4.  Modify  visual  attributes  of  the  data  (e.g.  changing  the  transparency  of  a  curve): 

5.  Add  animation  to  show  uncertainty  (e.g.  blinking  curves) 

6.  Add  acoustic  cues  to  indicate  uncertainty 

7.  Add  psycho-visual  cues  to  indicate  uncertainty  (e.g.  subliminal  images  could  be 
show  to  provide  alternative,  but  less  likely,  interpretations). 

Davis  and  Keller  (1997)  suggested  that  representation  techniques  could  be  broadly 
categorised  as  either  static  or  dynamic  techniques.  For  example,  the  use  of  animation 
would  be  considered  a  dynamic  technique,  while  the  other  categories  would  be  considered 
as  static  techniques. 

3.2  Uncertainty  in  geospatial  contexts 

Griethe  and  Schumann  (2006)  suggested  five  simplified  and  intuitive  categories  of 
visualisation  for  uncertainty  in  geospatial  contexts  as  described  in  the  following  sections. 

3.2.1  Utilisation  of  graphical  variables 

32.1.1  Colour 

There  have  been  many  visualisation  approaches  described  that  use  the  HSV  (Hue, 
Saturation,  Value)  components  of  colour  to  display  uncertainty  (See  Figure  4).  MacEachren 
(1992)  considered  colour  saturation  as  "the  most  logical  one  to  use  for  depicting 
uncertainty".  He  argued  that  information  with  a  high  level  of  certainty  should  be 
represented  by  pure  hues,  while  less  certain  information  should  use  a  correspondingly  less 
saturated  colour  (i.e.  greying  out  uncertain  areas  to  make  their  colour  hue  "uncertain"). 
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However,  as  illustrated  in  Figure  4,  the  use  of  colour  saturation  has  possible  limitations  in 
that  colours  with  low  saturation  may  be  difficult  to  distinguish  from  each  other  (Brown 
and  van  Elzakker,  1993). 


Figure  4  :  The  HSV  cone  from  (Potter,  2010)  shows  how  it  is  difficult  to  distinguish  between  low 
saturation  colours. 

Uncertainty  can  also  be  represented  using  the  hue  (H)  component  of  a  visualisation  (See 
Figure  5).  For  example,  a  red  hue  could  be  used  to  represent  higher  uncertainty  in  a 
measurement,  while  blue  could  be  used  to  represent  lower  uncertainty  (Howard  and 
MacEachren,  1996).  A  related  approach  is  to  simply  use  colour  coding  of  data  features  to 
represent  different  levels  of  uncertainty  (Appleton,  2004). 


Figure  5:  (a)  The  usage  of  colour  (hue)  to  encode  uncertainty  in  the  future  development  of  a  rural 
environment  setting  (Appleton,  2004)  (b)  Usage  of  bivariate  representations  that  depict 
data  (inorganic  nitrogen  in  Chesapeake  Bay)  and  uncertainty  together  using  colour 
value  (dark  means  more  nitrogen)  and  hue  (blue  means  more  certain) (Howard  and 
MacEachren,  1996) 
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Jiang  et  al.  (1995)  proposed  a  different  HSV  encoding  system  to  represent  uncertainty  in 
fuzzy  spatial  analysis.  In  this  approach,  hue  is  used  to  assign  nominal  categories  while 
saturation  can  confer  data  values,  and  lightness  is  changed  to  show  uncertainty.  (Dark 
areas  have  a  higher  certainty  than  light  areas.) 

MacEachren  (1992)  also  considered  that  texture  is  the  most  appropriate  approach  to  depict 
whether  information  is  "certain  enough"  or  "not  certain  enough".  Davis  and  Keller  (1997) 
suggested  that  colour  hue,  colour  value,  and  texture  are  potentially  the  best  choices  for 
representing  uncertain  information  using  static  techniques. 

3.2.1.2  Focusing  metaphor 

MacEachren  (1992)  proposed  a  "focusing"  metaphor  that  uses  "out  of  focus"  depictions  for 
uncertain  information  and  "in  focus"  depictions  for  certain  information.  He  suggested 
using  three  graphical  variables  to  depict  uncertainty  in  this  way:  crispness,  resolution  and 
transparency.  Crispness  describes  how  easily  different  areas  of  the  graphic  can  be 
distinguished,  as  shown  in  Figure  6(a)  and  6(b).  Resolution  describes  the  significance  of 
different  features  of  the  graphic,  as  illustrated  by  the  different  levels  of  pixelation  shown 
in  Figure  6(d).  In  his  usage,  transparency  represents  the  level  of  fogginess  applied  to  the 
graphic.  A  higher  degree  of  fogginess  is  used  to  represent  higher  levels  of  uncertainty,  as 
shown  in  Figure  6(c). 


(a) 


4**4 


**** 

**** 

**** 

**** 

**** 


(b) 


Figure  6:  Uncertainty  represented  using  different  graphical  variables  ( MacEachren ,  1992):  (a)  The 
depiction  of  a  risk  zone  around  a  nuclear  power  plant  showing  certainty  vs  uncertainty 
using  contour  crispness;  (b)  The  depiction  of  land  cover  type  showing  certainty  vs 
uncertainty  using  less  crisp  symbols  or  glyphs;  (c)  The  depiction  of  ecological  risk  due  to 
ozone  showing  certainty  vs  uncertainty  using  levels  of  fogginess  ('transparency');  (d) 
The  depiction  of  geography  showing  certainty  vs  uncertainty  using  pixilation. 
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Similarly,  Brown  (2004)  argued  that  mapping  the  uncertainty  to  a  colour  map  as  a  feature 
does  not  produce  an  immediate  "perception"  of  uncertainty  in  the  data  and  it  has  to  be 
"interpreted"  via  a  higher  level  cognitive  process.  He  suggested  that  "blurring"  is  the  most 
immediate  and  intuitive  metaphor  for  depicting  uncertainty.  As  illustrated  in  Figure  7, 
blurring  is  the  attenuation  of  the  high  spatial  frequency  details  of  the  visualisation.  By 
removal  of  the  high  frequency  information  in  the  visualisation,  the  viewer  recognises  the 
lowered  confidence  in  that  information. 
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§1  A 

AAA  A 


4 

zaZ  |  a  5 

□  ft  o&e 

Figure  7:  A  chess  tutoring  system  showing  the  chessmen  that  threaten  (left)  and  cover  (right)  the 
knight  on  e3  (Kosara  et  al,  2001)  using  blurring.  This  effect  could  also  be  used  to  blur 
uncertain  information  in  a  visualisation. 

Pang  et  al.  (1997)  suggested  the  use  of  a  stereoscopic  effect  to  generate  blurring.  With  3D 
stereo  glasses,  a  3D  stereo  effect  is  achieved  by  alternating  the  left  and  right  images  every 
frame  to  the  left  and  right  eyes.  Blurriness  could  be  achieved  by  showing  two  different 
images,  or  two  identical  images  could  be  used  to  display  a  clear  picture. 


3. 2. 1.3  Opacity 

In  contrast  to  the  "fog"  effect  in  Figure  6(c),  Drecki  (2002)  proposed  an  "opacity"  method 
which  utilised  the  transparency  of  the  object  to  depict  uncertainty.  In  this  method,  the 
highly  transparent  objects  indicate  high  uncertainty,  while  opaque  objects  are  considered 
certain. 


Figure  8:  The  usage  of  transparency  to  encode  uncertainty  in  medieval  architecture  (Isenberg,  1999) 
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Figure  8  illustrates  a  graphical  reconstruction  of  a  medieval  building,  depicting 
uncertainty  about  the  architecture.  While  the  foundations  of  the  building  are  known  and 
more  clearly  shown,  the  architecture  of  the  building  above  the  ground  is  less  certain  and 
so  is  represented  using  transparency. 

3.2.2  Integration  of  additional  objects 

32.2.1  Glyphs 

Another  approach  used  to  represent  uncertainty  is  to  map  it  to  additional  graphics  such  as 
glyphs,  labels,  contours,  isosurfaces,  etc.  Glyphs  are  graphical  objects  in  which  multiple 
visual  variables  can  be  used  to  represent  multiple  data  attributes.  They  are  thus  useful  for 
representing  uncertainty,  especially  multiple  types  of  uncertainty  (Pang,  2001, 
MacEachren  et  al.,  2005,  Sanyal  et  al.,  2009).  For  example,  to  visualise  uncertainty  in  wind 
and  ocean  currents.  Pang  (2001)  represented  uncertainty  in  direction  and  magnitude  using 
different  glyph  attributes  (see  Figure  9).  Pang  (2001)  also  suggested  that  different  types  of 
glyph  can  be  used  to  represent  varying  levels  of  uncertainty. 


Figure  9:  Glyphs  indicating  wind  direction,  magnitude  and  uncertainty.  The  width  of  the  glyph 
head  corresponds  to  angular  uncertainty.  (Pang,  2001) 
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3.2.22  Contours 

Contour  lines  are  a  two  dimensional  visualisation  technique  widely  used  to  show  lines  of 
equal  value  for  a  variable  in  three  dimensional  data.  Uncertainty  in  contours  can  be 
represented  by  a  variety  of  techniques.  For  example:  by  modifying  contour  colour  (Osorio 
and  Brodlie,  2008);  widths  (Dutton,  1992);  and  opacity  (Zhang  et  al.,  2010).  In  addition, 
spatial  uncertainty  can  be  encoded  as  the  size  of  gaps  in  broken  contour  lines  -  the  more 
uncertain,  the  larger  the  gaps  (Pang,  2001). 


Figure  10:  Example  of  uncertainty  representation  using  broken  contours.  The  larger  gaps  mean 
higher  uncertainty  (Pang,  2001) 

32.2.3  Isosurface  rendering 

An  isosurface  is  a  three  dimensional  analogue  of  a  contour  line  (or  isoline)  which 
represents  surfaces  of  constant  value  within  a  volume  of  space.  The  conventional 
isosurface  rendering  applies  a  constant  colour  to  isosurfaces,  allowing  different  colours 
and  textures  to  be  used  to  represent  additional  information.  For  example,  Rhodes  et  al. 
(2003)  proposed  uncertainty  visualisation  for  isosurfaces  using  an  additional  texture 
mapped  on  top  of  the  surface,  with  variations  of  hue  and  opacity  of  the  texture  to 
represent  the  level  of  uncertainty  (see  Figure  11). 
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Figure  11:  Different  degrees  of  uncertainty  in  a  medical  domain  are  shown  here  using  hue  and 
texture,  (a)  Colour  key  showing  example  of  A:  Hue,  B:  Texture  C:  Hue+Texture  (b) 
Uncertainty  value  mapped  to  texture  opacity  (with  constant  hue)  (c)  Uncertainty  value 
mapped  to  hue  and  texture  (Rhodes  et  al,  2003) 


3.2.3  Animation 

There  have  been  many  approaches  which  use  animation  to  represent  uncertainty.  One 
way  to  represent  uncertainty  is  to  allow  two  values  to  exist  in  the  same  spatial  location, 
with  an  animation  oscillating  between  the  two  values.  This  provides  a  temporal  mapping 
to  uncertainty  that  could  be  used  to  show  the  range  of  possible  values  for  data 
representing  a  continuous  variable.  Or,  for  categorical  data,  the  degree  of  uncertainty  can 
be  depicted  by  changing  the  animation  parameters  such  as  duration,  speed,  rate  of  change, 
order  and  range  or  extent  of  motion.  Fisher  (1993)  presented  an  "error  animation" 
technique  using  a  complex  blinking  effect.  In  his  example,  soil  grids  could  belong  to  one  of 
several  possible  soil  classes  represented  by  a  particular  colour.  An  animation  was  used 
where  the  colour  of  the  soil  grid  rotated  between  the  possible  classes,  with  the  dwell  time 
mapping  to  the  likelihood  of  that  class.  As  illustrated  in  Figure  12,  Brown  (2004) 
implemented  several  uncertainty  visualisation  techniques  for  3D  terrain  data,  animating 
vertex  height,  surface  luminance  and  hue. 
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(a)  Frames  from  a  movie  of  the  vertex  oscillation  technique,  with  the  changes  in  height  values 


(b)  Frames  from  a  movie  of  the  luminance  oscillation  technique 


Figure  12:  Uncertainty  representations  by  the  use  of  animation  (Brown,  2004). 


3.2.4  Non-visual  techniques 

One  limitation  of  representing  uncertainty  visually  is  that  it  increases  the  dimensionality 
of  the  data,  the  richness  needed  in  the  interface  (e.g.  colour  vs  monochrome),  and  the 
visual  complexity  of  the  display.  There  have  been  many  approaches  using  other  modalities 
such  as  auditory  and  haptic  cues.  These  avoid  cognitive  overload  of  the  visual  processing 
channel  by  using  additional  cognitive  channels  to  interface  with  the  computer  system. 
Such  multimodal  systems  have  also  been  used  to  support  attention  management  in  data- 
rich  environments  where  operators  face  considerable  visual  attentional  demands  (Sarter, 
2006).  For  effective  information  delivery,  these  multimodal  systems  should  be  designed 
carefully  with  consideration  of  the  user's  cognitive  system  (Mayer  and  Moreno,  2002, 
Sorden,  2005). 
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3.2. 4.1  Acoustic 

Krygier  (1994)  suggested  that  uncertainty  could  be  provided  by  incorporation  of  acoustic 
displays.  For  example,  a  sound  map  can  be  implemented  to  complement  the  existing 
visual  map.  The  degree  of  uncertainty  can  be  represented  by  a  variable  pitch,  volume  or 
rhythm  that  can  be  accessed  by  mouse  interaction  with  the  visual  display,  such  as  clicking 
and  dragging.  This  presentation  technique  could  be  a  solution  to  the  problem  of 
representing  and  comprehending  uncertainty  in  an  already  crowded  visual  display. 

Lodha  et  al.  (1996)  also  introduced  a  system  which  allows  the  user  to  explore  the 
dataspace  using  sonification  techniques  similar  to  those  described  above.  However,  a 
limitation  of  acoustic  representation  of  uncertainty  in  this  way  is  that  it  provides  feedback 
on  uncertainty  at  a  point  only,  whereas  a  visual  display  gives  a  global  view  (Brodlie  et  al., 
2012). 

3.2.42  Haptic 

Likewise,  the  use  of  haptic  senses  (e.g.  touch  or  vibration)  as  additional  modalities  can 
enhance  users'  abilities  to  process  more  information  simultaneously.  Schmidt  et  al.(2004) 
combined  haptic  and  visual  approaches  by  mapping  uncertainty  to  the  haptic  channel  to 
complement  a  visual  representation  using  glyphs.  In  this  approach,  they  mapped  the 
degree  of  uncertainty  to  the  glyphs'  degree  of  stiffness  in  a  haptic  channel.  In  this  case, 
haptic  devices  (Immersion©'s  CyberForce  Grasp  system)  produced  greater  force  feedback 
to  user's  fingers  for  larger  values  of  uncertainty  (see  Figure  13).  Since  they  applied  the 
haptic  channel  only  when  necessary,  this  approach  provides  an  intuitive  understanding  as 
to  where  the  most  prominent  uncertainty  lies  in  the  data. 


Figure  13:  Haptic  representation  of  uncertainty  uses  stiffness  of  the  glyphs  to  convey  uncertainty 
(Schmidt  et  at.,  2004) 
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3.2.5  Interactive  representation 

Uncertainty  can  also  be  provided  by  other  forms  of  user  interaction.  Wei  et  al.  (1997) 
proposed  a  "clickable  map"  where  information  concerning  the  uncertainty  is  not  directly 
visible  in  the  map  but  may  be  activated  by  pressing  the  mouse  button  at  a  pixel,  providing 
a  'local'  view  of  the  data  in  a  similar  way  to  sonification  and  haptic  feedback. 


3.3  Evaluation  of  uncertainty  visualisation  techniques 

User  evaluations  provide  a  systematic  approach  to  understanding  the  usability  of 
techniques  and  manipulable  interfaces  for  using  those  representations  (MacEachren  et  al., 
2005).  While  most  studies  on  uncertainty  visualisation  have  focused  on  developing  new 
technical  applications  and  algorithms,  little  evaluation  has  been  done  on  the  effectiveness 
of  their  approaches.  To  empirically  assess  the  effectiveness  of  proposed  visualisation 
techniques,  user  studies  with  relevant  participants  should  be  included.  Riveiro  (2007) 
argued  that  most  uncertainty  visualisation  research  does  not  include  a  perceptual  and/  or 
cognitive  analysis,  or  user  evaluations  that  validate  the  usefulness  of  the  proposed 
approaches.  Similarly,  Tory  and  Moller  (2004)  argued  that  only  a  few  information 
visualisation  designs  utilised  perceptual  and  cognitive  theories,  and  this  makes  it  difficult 
to  systematically  choose  promising  ideas  for  further  study. 

Kinkeldey  et  al.(2014),  reviewed  some  of  the  evaluations  of  uncertainty  visualisation 
techniques  and  grouped  them  into  two  categories:  objective  assessment  and  subjective 
assessment.  Objective  assessment  includes  tasks  with  measurable  correctness  of  results 
such  as  value  retrieval,  ratings,  comparisons  or  rankings.  Whereas,  for  instance,  subjective 
assessment  includes:  evaluation  of  the  intuitiveness  of  an  approach;  the  preference 
compared  to  other  options;  or  the  subjects'  confidence  in  their  responses  when  using  it. 

The  following  chapter  discusses  some  of  the  relevant  user  studies  on  uncertainty 
visualisation  for  decision  making  tasks. 
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4.  Decision  Making  with  Uncertainty 


The  reliability  of  the  information  available  to  a  decision  maker  influences  the  reliability  of 
the  decisions  made.  Indication  of  uncertainty  in  the  visualisations  provided  to  a  decision 
maker  is  thus  crucial  to  decision  making  (Dei trick,  2007).  The  information  visualisation 
tools  used  by  decision  makers  should  make  them  aware  of  the  presence,  nature,  and 
degree  of  uncertainty  in  the  data,  otherwise  they  may  draw  inaccurate  conclusions,  which 
could  lead  to  costly  mistakes  (Olston  and  Mackinlay,  2002).  Indeed,  uncertainty 
visualisation  has  long  been  regarded  as  a  difficult  topic  in  C2  since  the  commander  has  to 
make  decisions  in  a  limited  time  frame  with  information  that  comes  from  varied  sources, 
in  many  formats,  with  a  continuum  of  validity  (Summers  et  al.,  2005). 

The  reliability  of  C2  decision  making  is  affected  by  uncertainties  in  the  sensor  data,  the 
possible  interpretations  of  the  situation  that  is  developing,  and  on  the  degrees  of  freedom 
in  the  models  used  to  evaluate  the  impact  on  the  commander's  intent.  More  work  also 
needs  to  be  done  on  the  link  between  uncertainty  and  risk3,  such  as  investigating  the 
influence  of  the  uncertainty  associated  with  the  input  variables  of  a  model  on  the  risk 
associated  with  decision-making  (Brodlie  et  al.,  2012). 
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Figure  14:  A  hasty  attack  across  a  water  obstacle  requires  complex  reasoning  about  temporal, 
spatial,  and  categorical  uncertainties  (James  et  al,  1999). 


3  According  to  NATO  Code  of  Best  practice  (NATO  2004.  NATO  Code  of  Best  Practice  for 
Command  and  Control  Assessment  RTO  TECHNICAL  REPORT  TR-081),  "risk"  is  defined  as  the 
possibility  of  suffering  harm  or  loss  and  "uncertainty"  is  defined  as  an  inability  to  determine  a 
variable  value  or  system  state,  or  predict  its  future  evolution 
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4.1  Information  fusion  systems  with  uncertainty 

Commanders  are  expected  to  maintain  situation  awareness  of  the  battle  space  so  they  can 
make  time  critical  decisions  in  a  complex,  uncertain,  and  ever-changing  environment. 
They  are  increasingly  being  required  to  conduct  new  types  of  missions  and  operations 
other  than  war.  These  require  a  wide  diversity  of  information,  including  troop  positions, 
readiness  data,  human  intelligence  reports,  cultural  factors  and  political  drivers.  To 
successfully  conduct  a  campaign,  commanders  need  to  assimilate  large  amounts  of  data, 
from  a  wide  variety  of  sources,  in  limited  timeframes.  Yu  et  al.(2004)  argued  that  attempts 
to  simply  bring  more  information  to  commanders  are  doomed  to  failure  due  to  cognitive 
overload.  As  discussed  previously,  displaying  massive  amounts  of  data  without 
consideration  of  human-computer  interface  (HCI)  efficacy  can  potentially  obscure  critical 
information  and  degrade  the  effectiveness  of  the  system  (Chambers  et  al.,  1983),  resulting 
in  loss  of  the  decision  maker's  situation  awareness. 


Situation  awareness  can  be  defined  as  'the  perception  of  the  elements  in  the  environment  within 
a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status 
in  the  near  future'  (Endsley,  1995).  Information  fusion  systems  seek  to  improve  situation 
awareness  by  reducing  the  cognitive  load  on  the  decision  maker.  Information  fusion 
systems  automatically  or  semi-automatically  process  and  reason  about  large  volumes  of 
information  from  multiple  sources  (Bosse  et  al.,  2007,  Blasch  et  al.,  2012).  The  decision 
maker(s)  can  be  considered  to  be  an  integral  component  of  these  systems,  handling  those 
tasks  that  are  too  problematic  for  machines  (Lambert,  2009). 
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Figure  15:  A  schematic  vieiv  of  information  fusion  systems  (from  (Nilsson  et  al,  2008)) 
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The  standard  JDL  model  for  information  fusion  systems  conceptualises  information  fusion 
as  a  four4  level  process  in  which  different  types  of  information  are  processed  and  different 
types  of  outputs  are  obtained: 

•  Level  1:  Object  assessment,  in  which  sensor  detections  or  entity  features  are  fused  to 
estimate  the  state  of  an  object. 

•  Level  2:  Situation  assessment,  in  which  the  states  of  objects  are  used  to  estimate  the 
relations  between  them,  and  hence  the  evolving  state  of  a  situation. 

•  Level  3:  Impact  assessment,  in  which  the  evolving  situation  is  used  to  estimate  likely 
scenarios,  and  make  predictions  about  the  impact  of  these  scenarios  on  the  decision 
maker's  intent. 

•  Level  4:  Process  refinement,  in  which  the  outputs  of  a  processing  stage  are  used  to 
refine  the  estimation  process  at  one  or  more  levels.  This  is  not  a  separate  processing 
stage  per  se,  but  is  integrated  with  all  levels. 
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Figure  16:  Tire  standard  JDL  model  of  information  fusion  ( from  (Nilsson  et  ah,  2008)) 


A  key  element  of  information  fusion  systems  is  the  appropriate  representation  of  the 
information  used  to  guide  the  decision-making  processes  (Nilsson  et  al.,  2008).  A  single 
data  representation  is  not  appropriate  for  all  of  these  levels,  and  different  techniques  are 
needed  for  machine  processing  at  each  level.  As  discussed  in  Chapter  2,  some  degree  of 
uncertainty  is  almost  always  included  in  the  information  sources  and  the  processing 
stages.  In  this  case,  the  decision  maker  may  need  to  be  aware  of  the  nature  and  the  degree 
of  uncertainty  in  the  displayed  information  at  each  level,  and  a  different  representation 
may  be  required  for  this.  Summers  et  al.  (2005)  argued  that  situational  awareness,  and  the 
opportunity  for  decision  superiority,  is  increased  when  relevant  data  is  effectively 
presented  in  a  timely  manner  with  a  level  of  confidence.  Information  fusion  systems  which 
appropriately  convey  uncertainty,  particularly  its  degree  and  its  nature,  may  thus  be 
critical  to  allowing  the  decision  maker  to  comprehend  the  situation  and  project 
consequences  into  the  future  (Riveiro,  2007). 


4  A  5th  'Level  0'  fusion  stage  is  often  included  in  which  sensor  (or  other)  signals  are  fused  to  produce 
a  sensor  detection  above  some  background  or  noise  threshold,  but  this  level  of  processing  is  rarely 
displayed  to  an  operational  decision  maker. 
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4.2  User  studies 

As  discussed  in  §3.3,  only  a  limited  number  of  studies  have  been  done  on  the  effectiveness 
of  different  approaches  for  uncertainty  visualisation  in  decision  making.  This  section 
discusses  the  impact  of  uncertainty,  different  uncertainty  representation  techniques,  and 
user  experience  on  decision  making. 

4.2.1  The  effect  of  uncertainty  on  situation  awareness 

John  et  al.  (2000)  conducted  two  experiments  to  investigate  the  effect  of  uncertainty  on 
battlefield  situation  awareness  and  decision  making.  Their  first  experiment  examined  how 
different  representations  concerning  the  enemy's  intent  affect  the  situation  awareness  in  a 
ground  battlefield  situation.  Based  on  the  assumption  that  better  situation  awareness  will 
lead  to  better  understanding  and  recall  of  the  enemy's  location  and  improve  predictions  of 
their  future  movements,  experiments  were  conducted  with  marines  who  had  experience 
working  in  the  combat  operation  centre  (COC).  They  showed  that  two  graphical 
representations  (arrows  and  blobs  -  see  Figure  17)  of  enemy  intent  provided  significantly 
better  recall  of  relative  direction  and  distance  of  enemy  future  position  than  the  baseline 
without  any  representation,  which  required  them  to  infer  this  from  other  information. 


Figure  17:  Graphical  representations  of  uncertainty  in  future  enemy  position  (a)  blobs:  enemy's 
expected  location  in  6  hours  in  a  boundary  (b)  arroivs:  expected  movement  of  enemy  in 
future  (in  2,  4  and  6  hours) 

Their  second  experiment  explored  the  effect  of  situation  uncertainty  on  the  time  required 
to  make  a  decision.  Three  different  levels  of  uncertainty  concerning  enemy's  position  and 
capabilities  were  applied  to  tactical  decision  games.  Participants  were  divided  into  two 
groups  according  to  their  level  of  tactical  decision  making  experience.  The  results  revealed 
that  the  decision  times  for  experienced  participants  were  not  affected  by  the  degree  of 
uncertainty  concerning  enemy  strength  and  position;  however  inexperienced  participants 
were  more  likely  to  choose  to  wait  before  acting  when  uncertainty  was  high. 
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This  conclusion  was  validated  by  Kobus  et  al.  (2001).  In  this  study,  uncertainty  was  not 
visually  represented  but  was  introduced  through  ambiguity  in  scenario.  Their 
experimental  subjects  consisted  of  52  marines  who  had  high  or  low  command  post 
experience.  They  were  split  equally  into  two  groups  that  participated  in  a  virtual 
command  post  scenario  that  depicted  a  land  warfare  engagement  and  battle.  One  group 
began  the  scenario  early,  when  uncertainty  in  the  situation  was  high  and  reconnaissance 
reports  were  not  yet  available,  while  the  other  group  began  the  scenario  later  when 
uncertainty  was  lower  but  an  engagement  was  imminent  and  decision  making  time 
limited.  Thus,  this  experiment  investigated  decision  making  with  a  2  (uncertainty)  x  2 
(experience)  factorial  design  (N=52).  Their  results  revealed  that  the  experienced 
participants  were  faster  than  the  inexperienced  group  at  executing  a  course  of  action  early 
in  the  scenario  when  uncertainty  was  higher,  but  were  not  faster  than  the  inexperienced 
group  at  making  tactical  decisions  later  in  the  scenario  when  the  uncertainty  was  lower 
and  time  pressures  were  higher.  However,  perhaps  not  surprisingly,  the  more  experienced 
participants  were  significantly  more  accurate  than  the  less  experienced  participants  at 
developing  an  appropriate  course  of  action. 

4.2.2  The  effect  of  colour  on  decision  making 

Leitner  and  Buttenfield  (2000)  conducted  an  experiment  that  explored  the  effects  of  colour 
value,  saturation  and  texture  as  the  means  of  depicting  uncertainty  on  maps.  He  looked  at 
the  time  (response  time  for  selection),  accuracy  (the  correctness  of  location),  and 
confidence  (confidence  level  in  the  decision)  of  users  of  a  spatial  decision  support  system. 
Participants  were  asked  to  use  a  map  to  make  site  decisions  for  a  conservation  area  (easy 
decision)  and  an  airport  (difficult  decision)  in  wetlands,  based  on  a  set  of  predetermined 
planning  criteria.  Their  results  were  compared  with  the  site  locations  chosen  by  subject 
matter  experts.  The  authors  found  that  the  addition  of  uncertainty  information  using 
texture  (finer  texture  mapped  to  less  uncertainty)  or  colour  value  (lighter  colour  mapped 
to  less  uncertainty)  significantly  improved  performance  for  the  easy  decision,  but  no  such 
difference  was  observed  for  the  difficult  decision.  This  may  indicate  that  for  decision 
making,  the  difficulty  of  the  task  could  be  the  dominant  factor  compared  to  the 
uncertainty  representation  technique. 

The  user's  response  time  and  confidence  in  the  decision  made  seemed  to  be  similar  with 
and  without  uncertainty  representation.  From  this  the  authors  conjectured  that  the 
techniques  used  for  visualisation  of  uncertainty  did  not,  on  balance,  clutter  the  map 
display. 

Colour  saturation  was  not  found  to  be  particularly  effective  in  this  study,  which  is 
consistent  with  other  studies  (Drecki,  2002,  Sanyal  et  al.,  2009,  MacEachren  et  al.,  2012).  In 
particular,  transparency  has  been  found  to  be  a  more  effective  method  of  uncertainty 
representation  than  colour  saturation  (Newman  and  Lee,  2004).  However  users  have  been 
found  to  still  have  a  preference  for  colour  saturation,  regardless  of  its  low  performance 
(Drecki,  2002). 
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4.2.3  The  effect  of  animation  on  decision  making 

Aerts  et  al.  (2003)  implemented  SLEUTH  (Slope,  Land  use.  Exclusion,  Urban  extent. 
Transportation,  and  Hill  shade),  an  urban  growth  model  designed  to  aid  land  use 
planning.  SLEUTH  used  two  uncertainty  visualization  techniques:  static  comparison  and 
toggling.  In  their  static  comparison  approach,  a  model  of  the  result  and  its  uncertainty 
were  presented  in  side-by-side  images,  whereas  in  their  toggled  approach  the  result  and 
its  variations  were  sequenced  in  an  animated  loop  (four  frames  per  second).  Participants 
showed  a  significantly  higher  confidence  in  their  uncertainty  estimation  with  the  static 
representation  technique  than  with  the  animation  technique.  They  also  showed  a 
preference  for  the  static  technique.  Interestingly,  this  finding  seems  to  be  supported  by 
other  studies  that  showed  that  dynamic  techniques  were  less  effective  than  static 
representations  (Blenkinsop  et  al.,  2000,  Drecki,  2002). 


Figure  18:  Uncertainty  visualisation  in  SLEUTH  using  colour  value  (lighter  means  less  certain 
area) 

4.2.4  The  effect  of  extrinsic  representation  (glyph) 

Slocum  (2003)  studied  the  usability  of  their  software  tool  for  visualising  future  global 
water  balance,  which  is  a  function  of  many  uncertain  factors.  Two  visualisation 
approaches  for  uncertainty  were  applied  in  their  tool:  intrinsic  techniques  using  RGB  and 
colour  value;  and  extrinsic  techniques  using  glyphs  (see  Figure  19). 
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Figure  19:  Visualising  uncertainty  of  the  future  global  water  balance  (a)  Three-dimensional 
representation  of  surpluses  or  deficits  in  the  model  using  RGB  (R:  temperature,  G:  Soil 
and  B:  Precipitation)  with  the  colour  value  depicting  uncertainty,  (b)  Usage  of  Glyphs 
to  represent  uncertainty  associated  with  data  (Slocum  et  al,  2003) 


In  developing  the  software  tool,  they  utilised  the  principles  of  user-centred  design  and 
evaluation  (Gabbard  et  al.,  1999)  which  considered  the  importance  of  usability  testing  with 
actual  task  users  during  the  design  phase.  Thus,  during  their  software  development,  they 
conducted  user  tests  with  two  expert  groups:  decision  makers  consisting  of  employees  of 
the  state  government  agricultural  and  water  management  offices;  and  subject  matter 
experts  consisting  of  civil  engineers,  atmospheric  scientists,  and  geographers  who  were 
experts  in  water  balance  models. 

User  tests  showed  that  the  decision  makers  preferred  the  use  of  colour  to  depict 
uncertainty,  whereas  the  subject  matter  experts  preferred  the  use  of  glyphs.  The  authors 
conjectured  that  the  intrinsic  representation  gives  a  better  overview  to  allow  decision 
makers  to  more  easily  see  the  'big  picture',  and  that  the  extrinsic  representation  provides 
better  support  for  domain  experts  to  do  in-depth  analysis.  Based  on  these  results,  it  can  be 
concluded  that  uncertainty  visualisation  techniques  should  be  adjusted  to  suit  the  task. 

4.2.5  The  effect  of  iconic  representations  on  decision  making  for  target 
classification 

Finger  and  Bisantz  et  al.(2002)  evaluated  a  variety  of  iconic  representation  techniques  for 
visualising  uncertainty  in  decision  making  tasks.  In  their  first  study  they  showed  that 
subjects  could  appropriately  classify  five  sets  of  icons  to  represent  friendly  or  hostile 
targets,  as  shown  in  Figure  20. 
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Figure  20:  Icons  representing  object  identity  (hostile  or  friendly)  (Bisantz  et  al,  1999) 

They  then  showed  that  subjects  were  able  to  understand  different  levels  of  uncertainty 
associated  with  the  identity  of  radar  contacts  as  hostile  or  friendly  using  a  series  of  13 
degraded  and  blended  icons,  as  illustrated  in  Figure  21. 
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Figure  21:  Icons  representing  a  range  of  probabilities  (hostile  or  friendly).  The  numbers  indicate  the 
probability  of  friendly  (Finger  and  Bisantz,  2002) 

They  also  looked  at  the  use  of  degraded  icons  combined  with  a  numerical  probability 
estimate,  as  shown  in  Figure  22,  and  found  that  participants  using  only  degraded  icons 
show  significantly  better  decision  making  performance  than  the  other  options. 
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Figure  22:  Example  display  stimuli  for  the  degraded  only,  degraded  with  probability,  and 
probability  only  display  conditions  (Finger  and  Bisantz,  2002) 


4.2.6  The  effect  of  uncertainty  visualisation  on  target  identification 

Riveiro  et  al.  (2014)  presented  an  empirical  study  that  investigated  the  effects  of 
uncertainty  visualisation  on  decision  making  in  an  air  defence  scenario  where  decision 
time  is  limited.  As  illustrated  in  Figure  23,  they  adopted  semi-transparent  filled  circles  to 
represent  the  uncertain  position  of  tracks,  and  the  thickness  of  the  lines  for  the  accuracy  of 
scalar  values  such  as  altitude  and  range.  In  their  experiment,  there  were  119  individual 
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targets  that  required  identification.  Twenty  two  experienced  air  traffic  operators,  who 
were  divided  into  two  groups,  with  or  without  uncertainty  visualisation,  were  asked  to 
identify  and  prioritise  targets  that  they  thought  were  interesting  or  threatening.  The 
results  show  that  the  group  aided  by  uncertainty  visualisations  needed  significantly  fewer 
attempts  to  make  a  final  identification  and  identified  more  hostile  and  suspect  targets. 
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Figure  23:  Uncertainty  visualisation  (Riveiro  et  al,  2014) 


4.2.7  The  perceived  trust  level  of  information  in  decision  making 

When  decision  support  systems  need  to  present  statements  about  a  situation,  the 
uncertainty  associated  with  such  statements  can  be  conveyed  by  using  grading  descriptors 
(e.g.  unlikely,  possible,  probable  and  certain).  However,  these  qualifiers  could  be  easily 
ignored,  or  they  may  impose  an  additional  cognitive  burden  on  the  decision  maker.  Smith 
et  al.  (2005)  argued  that  the  level  of  uncertainty  can  be  conveyed  by  presenting  the 
information  using  an  appropriately  trustworthy  virtual  avatar.  They  explored  how  the 
perceived  trustworthiness  of  the  avatar  depends  on  facial  attributes  such  as  eye  size,  pupil 
size,  face  shape,  forehead  size,  eye  gaze  and  emotional  expression  (smiling,  neutral  and 
frowning)  (see  Figure  24).  In  this  study,  participants  were  asked  to  view  all  faces  and 
assess  the  trustworthiness  of  each  face  on  a  seven-point  scale  ranging  from  "completely 
trustworthy" (1)  to  "completely  untrustworthy" (7).  Their  results  show  that  the  facial 
characteristics  of  the  avatar  could  potentially  be  used  to  control  the  confidence  associated 
with  the  information  delivered  by  an  avatar. 

In  a  follow-up  experiment,  participants  were  asked  to  rate  their  confidence  in  the  advice 
from  a  'trustworthy'  avatar  with  the  same  advice  presented  as  simple  text.  In  this  case 
participants  considered  that  simple  text  is  a  more  reliable  source  of  information  than  an 
avatar.  Based  on  these  findings,  the  authors  argued  that  different  presentation  modes 
could  be  used  to  appropriately  convey  different  levels  of  reliability  in  the  information 
presented. 
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Figure  24:  Examples  of  stimulus  faces  with  trustworthy  scores  (lower  score  means  higher 
trustworthy  and  vice  versa) (Smith  et  al,  2005) 


4.3  User  issues 

Empirical  evaluation  of  uncertainty  visualisation  techniques  is  crucial  to  understanding 
the  effectiveness  of  these  techniques.  Most  of  all,  it  is  important  to  recruit  relevant 
participants  who  are  representative  of  the  target  user  group  in  terms  of  age,  skills, 
experience  and  so  on.  This  is  verified  by  studies  which  have  considered  the  participants' 
level  of  experience  (Kobus  et  al.,  2001,  Aerts  et  al.,  2003)  or  domain  of  expertise  (Slocum  et 
al.,  2003,  Kobus  et  al.,  2001,  Aerts  et  al.,  2003)  as  independent  variables. 

Similarly,  Kinkeldey  (2014)  argued  that  many  studies  predominately  recruited  university 
students  for  their  empirical  studies  due  to  their  ready  availability,  and  they  often  regarded 
students  as  domain  experts.  The  author  suggested  that  evaluation  studies  should  provide 
additional  training  to  allow  unexperienced  participants  to  gain  experience  with  the 
scenarios,  data,  tasks  or  visualisation  techniques  for  more  effective  empirical  studies. 
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5.  Guidelines  for  Uncertainty  Visualisation 

5.1  Graphical  principles  for  visualisation 

Several  authors  have  proposed  guidelines  for  data  visualisation  that  encompass  the 
requirements  for  uncertainty  visualisation.  For  example,  Tufte  (1983)  suggested  two 
graphical  principles  that  lead  to  good  visualisation: 

•  Graphical  excellence 

o  avoid  distorting  what  the  data  shows 

o  give  the  viewer  the  greatest  number  of  ideas  in  the  shortest  time  with  the 
least  "data-ink"5  in  the  smallest  space 

o  present  a  large  amount  of  data  in  a  small  space 

o  reveal  multiple  levels  of  detail 

o  closely  integrate  the  statistical  and  verbal  descriptions  of  a  data  set 
o  encourage  comparison  among  the  data. 

•  Graphical  integrity 

o  the  representation  of  numbers  should  be  directly  proportional  to  the 
numerical  quantities  represented 

o  clear,  detailed,  and  thorough  labelling  should  be  used  to  defeat  ambiguity 
and  distortion; 

o  show  data  variations  and  not  design  variations 

o  the  number  of  information-carrying  (variable)  dimensions  depicted  should 
not  exceed  the  number  of  dimensions  in  the  data 

o  convincing  graphics  should  demonstrate  cause  and  effect. 

Tufte  also  proposed  the  "data-ink  maximization  principle"  which  claims  that  good 
graphical  representations  maximise  data-ink  and  minimise  the  graphical  components  not 
conveying  data  directly  (e.g.  scales,  labels  and  edges). 

Summers  et  al.  (2005)  considered  five  key  techniques  for  display  management  to  increase 
situational  awareness  and  decision  superiority.  These  were  implemented  in  a  Tactical 
Ballistic  Missile  (TBM)  defence  prototype: 

•  Tailoring  the  display:  to  the  situation,  user's  role,  individual  user  preferences  and 
screen  size  (e.g.  hand-held,  desktop  or  data  wall). 

•  De-cluttering:  focus  on  important  information  by  providing  the  ability  to  filter  out 
detail  unnecessary  to  the  task  at  hand.  Techniques  should  interoperate  with  each 
other  in  a  dynamic  environment. 


5  Data-ink  refers  to  the  graphical  components  used  for  the  representation  of  data. 
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•  Integration:  fluidly  change  visualisation  techniques  as  the  situation  evolves. 

•  Cross  tool  control:  all  visualisation  tools  can  show  alternate  views  of  the  same 
situation,  or  views  can  be  set  and  controlled  by  one  for  the  others. 

•  Use  of  toolkits:  to  easily  extend  techniques  and  customized  displays  for  re-use. 


(a) 


(b) 


Figure  25:  TBM  prototype  implemented  using  five  key  display  management  techniques  (Summers 
et  ah,  2005):  (a)  Example  of  cluttered  screen  in  which  it  is  difficult  to  understand  the 
situation,  (b)  Reducing  clutter  through  overlay  filtering  makes  the  situation  easier  to 
comprehend. 

As  shown  in  Figure  25,  the  coverage  of  each  system  is  placed  on  a  separated  overlay  with 
different  colours.  Users  can  control  the  placement  of  each  overlay  on  the  display  through 
an  overlay  manager  which  can  create,  toggle  on/  off  and  reorder  individual  overlays.  By 
placing  the  display  of  these  overlays  under  user  control,  users  can  avoid  cluttering  the 
battlefield  map  with  unimportant  information  and  enhance  their  situational  awareness.  In 
this  example,  the  areas  of  coverage  on  an  overlay  are  drawn  as  semi-transparent.  From  a 
pragmatic  point  of  view,  the  author  suggests  that  usage  of  transparencies  for  overlays 
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provide  a  better  user  interface  but  renders  more  slowly  on  the  display  than  other  effects 
(e.g.  patterned  fills).6 

Interestingly,  Omodei  et  al.'s  (2005)  empirical  study  showed  that  decision  making 
performance  was  degraded  when  participants  were  provided  with  more  detailed 
information  compared  to  the  situation  in  which  they  were  presented  with  less  information. 
However,  they  assessed  themselves  as  performing  better.  The  authors  conjectured  that, 
under  time  pressure,  more  effort  was  spent  gathering  all  available  information  than 
paying  due  regard  to  the  important  information,  thus  degrading  their  decision  making 
ability.  This  result  suggests  that  caution  may  be  warranted  when  providing  users  with  the 
option  to  navigate  through  or  select  different  information  displays  or  layers,  including 
uncertainty  representations,  as  it  may  degrade  their  performance. 


5.2  Development  strategies  for  uncertainty  visualisation 

Lapinski  (2009)  proposed  an  Uncertainty  Visualisation  Development  Strategy  (UVDS) 
which  outlines  a  systematic  approach  with  eleven  strategies  for  designing  an  uncertainty 
visualisation,  in  the  context  their  research  supporting  the  Canadian  Recognized  Maritime 
Picture  (RMP). 
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Figure  26:  Unclassified  example  of  the  Canadian  Recognized  Maritime  Picture  (RMP)  off  the  coast 
of  Nova  Scotia,  Canada.  The  UVDS  was  applied  to  support  the  evolution  of  the  RMP 
(Lapinski,  2009) 


6  This  may  no  longer  be  a  concern  given  improvements  in  graphical  rendering  technologies. 
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The  stages  of  the  UVDS  are  as  follows  (note:  the  author  described  that  steps  4,  5,  6  and  7  in 
this  strategy  are  interconnected  and  can  therefore  be  completed  in  any  order): 

1.  Identify  the  uncertainty  visualisation  task.  This  provides  a  high  level  generalised 
description  of  the  task  that  can  be  phrased  as  a  problem/ mission  statement. 

2.  Understand  the  data  whose  uncertainty  needs  to  be  visualised.  This  ensures  that 
the  person  designing  the  visualisation  understands  the  primary  data  to  which  the 
uncertainty  is  related. 

3.  Understand  why  uncertainty  needs  to  be  visualised  and  how  the  uncertainty 
visualisation  needs  to  help  the  user.  The  intent  and  requirements  of  the  audience 
need  to  be  considered  since  the  same  data  may  need  to  be  visualised  differently  for 
different  audiences. 

4.  Decide  on  the  uncertainty  to  be  visualised.  Since  there  may  be  multiple 
uncertainties  associated  with  the  data,  the  designer  needs  to  recognise  the 
uncertainties  that  could  be  visualised  and  then  narrow  these  down  to  the 
uncertainties  that  should  be  visualised. 

5.  Decide  on  a  definition  of  uncertainty.  Increased  understanding  of  the  problem 
gained  from  previous  steps  will  allow  designers  to  define  more  appropriate 
definitions  for  the  task.  Lapinski  (2009)  reported  that  the  designers  feel  more 
comfortable  doing  this  step  before  step  4. 

6.  Determine  the  specific  causes  of  the  uncertainty.  This  step  illuminates  the  many 
causes  of  uncertainty  present  in  data  and  therefore  the  potential  causes  of  the 
uncertainties  the  designers  wish  to  focus  on,  which  is  important  for  the  designer  to 
understand. 

7.  Determine  the  causal  categories  of  the  uncertainty  to  illuminate  the  types  of 
uncertainty  the  designer  is  dealing  with.  This  information  will  be  valuable  when 
deciding  how  to  visualise  the  uncertainty.  Categories  of  uncertainty  are  related  to 
issues  of  timeliness,  precision,  availability  (missing  or  inaccessible),  error  (human 
or  machine),  and  ambiguity  (multiple  meanings  or  interpretations). 

8.  Determine  the  visualisation  requirements.  The  designer  identifies  the  visualisation 
needs  that  address  the  problem/ mission  statement. 

9.  Prepare  the  uncertainty  for  visualisation.  Do  what  is  necessary  to  have  the  required 
uncertainty  ready  to  be  used  in  the  visualisation.  This  could  involve  calculating 
uncertainty  from  collected  data,  transforming  uncertainty  measurements  into  the 
proper  units,  assigning  uncertainty,  extracting  uncertainty  from  collected  data,  etc. 

10.  Try  different  uncertainty  visualisation  techniques. 

11.  Seek  audience  opinions  and  criticisms. 

The  author  did  not  provide  an  empirical  evaluation  of  this  approach,  however  these 
guidelines  could  help  the  designer  better  understand  the  important  concepts  to  be 
considered  for  uncertainty  visualisation. 
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5.3  Challenges  for  uncertainty  visualisation 

In  MacEachren  et  al.  (2005),  the  authors  reviewed  several  visual  methods  and  tools  for 
uncertainty  visualisation  and  concluded  their  survey  paper  with  seven  core  challenges 
that  need  to  be  addressed  for  uncertainty  visualisation  research: 

1.  Understanding  the  components  of  uncertainty  and  their  relationships  to  domains, 
users,  and  information  needs 

2.  Understanding  how  the  knowledge  of  information  uncertainty  influences 
information  analysis,  decision  making,  and  decision  outcomes 

3.  Understanding  how  (or  whether)  uncertainty  visualisation  aids  exploratory 
analysis 

4.  Developing  methods  for  capturing  and  encoding  analysts'  or  decision  makers' 
uncertainty 

5.  Developing  representation  methods  for  depicting  multiple  kinds  of  uncertainty 

6.  Developing  methods  and  tools  for  interacting  with  uncertainty  depictions 

7.  Assessing  the  usability  and  utility  of  uncertainty  capture,  representation,  and 
interaction  methods  and  tools. 
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6.  Uncertainty  Visualisation  for  Joint  Operations 

Command 

6.1  Situational  awareness  for  JOC 

The  situational  awareness  requirements  of  the  Australian  Defence  Force's  Joint  Operations 
Command  (JOC)  range  across  the  spectrum  of  ADF  operations  to  achieve  national 
strategic  objectives.  The  users  in  JOC  need  to  access,  integrate  and  visualise  a  diversity  of 
information  across  the  Defence  Enterprise  including  military  text  messages,  unstructured 
documents,  military  databases  and  open-source  content  from  civilian  producers. 
Furthermore,  these  processes  need  to  be  tailored  to  support  a  variety  of  dynamic 
operational  needs,  because  the  users  are  unlikely  to  be  subject  matter  experts  in  each  of  the 
data  sets  being  integrated.  It  is  crucial  that  they  understand  the  uncertainties  associated 
with  the  information  being  displayed  so  that  they  can  mitigate  operational  risks. 

Some  specific  use-cases  of  relevance  to  coalition  operations  in  particular,  but  which  are 
representative  of  the  user  requirements  in  general,  are  discussed  below  (Read  et  al.,  2014). 

6.1.1  Disaster  response 

In  response  to  a  natural  or  other  disaster,  the  disposition  and  status  of  land,  air,  and 
maritime  forces,  the  effective  capability  they  can  bring  to  bear,  and  their  availability  to 
respond,  needs  to  be  established.  This  potentially  applies  to  not  only  formed  units  and 
deployed  forces,  but  contingency  forces  as  well.  In  addition,  situational  awareness  is 
required  of  the  disaster,  its  impacts  on  civilian  populations  and  infrastructure,  and  the 
capabilities  that  need  to  be  provided.  This  use-case  applies  at  a  national  and  an 
international  level  where  ad-hoc  coalitions,  including  defence  forces,  NGOs  and  OGAs, 
may  be  formed  and  reformed  as  required  to  meet  emerging  needs. 

The  types  of  information  sources  in  this  use-case  include: 

•  Open-source  -  freely  accessible  from  the  internet  but  often  poorly  controlled.  They 
can  be  provided  by  news  services,  government  departments  and  NGOs,  local 
governments,  social  media  or  other  crowd-sourced  content.  They  can,  for  example, 
contain  information  such  as  incident  locations  and  areas  of  effect,  geopolitical 
factors  (international,  national,  local  government),  weather,  civilian  infrastructure, 
social  networks  and  relationships. 

•  Diplomatic  -  usually  sourced  from  government  organisations  and  relatively  well 
controlled.  They  can,  for  example,  contain  foreign  policy,  relevant  treaties, 
protocols  for  dealing  with  NGOs,  and  international  political  considerations. 

•  Emergency  services  -  usually  sourced  from  government  or  affiliated  organisations 
and  relatively  well  controlled.  They  can  provide  information  about  emergency 
events  and  ongoing  crises,  available  resources,  capabilities  and  capability  gaps. 
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•  Legal  -  usually  sourced  from  government  organisations  and  well  controlled.  They 
can  provide  information  about  legal  requirements  and  constraints,  rights  of  access, 
legal  response  options  and  local  laws. 

•  Policy  -  usually  sourced  from  government  organisations  and  well  controlled.  They 
can  provide  information  about  operating  and  communication  protocols, 
information  sharing  constraints,  and  rules  of  behaviour. 

•  Preparedness  -  usually  sourced  from  defence  systems,  well  controlled  and 
generally  with  a  national  security  classification.  They  can  provide  information 
about  the  readiness  and  status  of  forces  and  their  assigned  roles. 

•  Logistics  -  usually  sourced  from  defence  systems,  well  controlled  and  generally 
with  a  national  security  classification.  They  can  provide  consumption  &  movement 
models  and  the  composition  of  formed  military  units. 

•  Capabilities  -  usually  sourced  from  defence  systems,  well  controlled  and  generally 
with  a  national  security  classification.  They  describe  the  capabilities  of  military 
units  and  forces. 

•  Operational  -  usually  sourced  from  defence  systems,  well  controlled  and  generally 
with  a  national  security  classification.  They  describe  the  current  tasking,  location, 
status  of  deployed  units  and  forces. 

•  ISR  (Intelligence,  Surveillance  and  Reconnaissance)  -  usually  sourced  from 
defence  systems,  well  controlled  and  generally  with  a  national  security 
classification.  In  this  use-case  they  would  provide  information  about  the 
disposition  of  national  and  coalition  assets,  civilian  agencies,  refugees  and  other 
civilians. 

6.1.2  Search  and  rescue 

This  use-case  has  similar  information  requirements  to  the  Disaster  Response  use-case 
above,  except  that  the  scope  of  the  activity  is  much  more  tightly  focussed,  usually  (but  not 
always)  on  the  recovery  of  a  single  group  of  people,  vessel,  or  aircraft. 

6.1.3  Maritime  domain  awareness 

In  this  use-case,  national  and  coalition  capabilities  may  be  involved  in  surveillance  of  sea- 
lanes  and  maritime  approaches  to  identify  unusual  or  suspect  behaviour.  This  may  be 
indicative  of  illegal  or  hostile  activities,  and  JOC  may  be  required  to  respond  appropriately. 
The  role  of  national  and  coalition  military  forces  in  this  use-case  may,  in  some  instances, 
be  to  support  the  civilian  law-enforcement  organisations  rather  than  take  a  lead  role,  and 
so  may  require  access  to  information  sources  that  are  not  managed  by  Defence. 

The  types  of  information  sources  required  for  this  use-case  are  similar  to  those  for  the 
other  use  cases,  and  also  include: 

•  Law-enforcement  services  -  usually  sourced  from  government  organisations  and 
well  controlled.  They  can  provide  information  about  the  status,  locations,  and 
capabilities  of  law  enforcement  agencies  and  assets; 
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•  Merchant  shipping  -  usually  sourced  from  commercial  organisations,  or  by  proxy 
from  law  enforcement  organisations.  They  can  provide  vessel  itineraries,  cargo  & 
crew  manifests  and  ports  of  call. 

6.2  SAKI 

SAKI  (Situational  Awareness  Knowledge  Infrastructure)  is  being  developed  by  DST 
Group  to  explore  the  challenges  of  integrating  information  from  diverse  operational 
sources,  and  refine  how  that  information  needs  to  be  displayed  to  meet  different 
operational  needs  and  roles.  It  allows  JOC  users  to  access  and  evaluate  prototype 
capabilities  from  their  desktops,  using  operational  data  sources.  SAKI  uses  a  layered 
service  architecture  as  shown  in  Figure  27. 
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Figure  27:  DST's  SAKI  system  is  being  developed  to  identify  and  refine  the  ongoing  services  and 
visualisation  requirements  of  Australia' s  Joint  Operations  Command  (Read  et  al,  2014) 


6.3  Sources  of  uncertainty  for  JOC 

The  sources  of  uncertainty  for  the  SAKI  system  can  be  categorised  using  the  3  stages 
discussed  in  §2.2  (Riveiro,  2007),  and  are  described  in  the  following  sections. 

6.3.1  Acquisition 

The  main  types  of  data  sources  used  by  SAKI,  and  indeed  JOC,  are  discussed  below.  While 
this  is  not  a  complete  list,  it  does  capture  the  main  classes  of  source  and  the  types  of 
uncertainties  associated  with  these  classes. 
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6.3.1.1  Military  Text  Format  (MTF)  messages 

Military  Text  Formatted  messages,  originally  developed  to  support  teletext  systems,  still 
form  the  bulk  of  current  military  reporting  systems,  although  they  are  currently 
transitioning  to  other,  more  flexible,  formats  such  as  XML.  For  this  discussion  we  will 
focus  on  manually  entered  reports  that  are  used  to  meet  a  variety  of  information 
requirements.  The  ADF  currently  uses  the  Mercury  system  to  send  these  messages,  which 
adds  a  header  that  provides  handling  and  delivery  instructions,  and  some  additional 
information: 

•  DTG  (Date  Time  Group):  specifies  when  the  message  was  sent. 

•  SIC  (Subject  Indicator  Code):  designates  the  operation  and  type  of  activity  the 
message  is  about. 

•  Subject:  A  free  text  field  that  should  provide  a  subject  line  for  the  message. 

The  body  of  the  message  contains  a  structured  set  of  fields  that  usually  include  a  free-text 
narrative  in  a  semi-structured  format,  written  by  humans  for  humans.  The  writers  make 
assumptions  about  common  knowledge  and  context.  This  free-text  narrative  generally 
needs  to  be  parsed  to  extract  the  information  needed  by  the  SAKI  system. 

The  types  of  messages  of  primary  interest  to  JOC  are: 

•  BBXX  voluntary  weather  observations  from  surface  vessels. 

•  COMREP  (Communications  Report):  contains  information  about  changes  to  the 
communications  circuits  and  networks. 

•  COMSTAT  (Communications  State):  contains  information  about  the 
communications  circuits  and  networks  to  be  used  for  the  next  24  hours. 

•  INCREP  (Incident  Report):  contains  information  about  particular  operational 
incidents,  such  as:  location,  time  of  incident,  personnel  involved,  number  of 
casualties,  number  of  civilians  involved,  etc.  Each  incident  report  represents  either 
a  new  incident  or  an  update  regarding  a  previous  incident. 

•  INTREP  (Intelligence  Report):  contains  updates  about  the  operating  environment 
and  actors  within  it,  normally  as  a  result  of  the  IPB  (Intelligence  Preparation  of  the 
Battlefield)  process. 

•  INTSUM  (Intelligence  Summary):  contains  a  daily  summary  of  the  operating 
environment  and  actors  within  it. 

•  JJVV:  contains  bathymetric  observations  from  surface  vessels. 

•  LOCATOR:  contains  a  positional  self-report  from  a  surface  vessel  at  sea.  Primarily 
contains  structured  data. 

•  LOCSTAT:  contains  the  current  location,  and  further  intention  or  direction  of  travel 
of  a  (usually  land)  unit. 

•  LOGSTAT  (Logistics  Status):  contains  a  report  on  the  logistics  status  of  a  unit.  This 
could  be,  for  example,  fuel  or  ammunition  remaining. 
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•  MISREP  (Mission  Report):  contains  updates  on  the  progress  or  status  of  a  mission 
assigned  to  a  unit. 

•  OPREP  (Operations  Report):  contains  updates  on  the  operational  activities  of  a  unit. 

•  OPSUM/OPSTAT  (Operations  Summary/ Status):  contains  a  daily  summary  of  the 
operational  status  and  capabilities  of  a  unit. 

•  SITREP  (Situation  Report):  contains  situation  updates  from/ for  a  unit. 

•  URDEF  (Urgent  Defect):  contains  a  report  on  operational  defects  for  a  unit  that 
may  impact  its  ability  to  carry  out  its  assigned  activities  or  mission. 


The  types  of  errors  that  can  occur  when  parsing  these  messages,  hence  leading  to 
ambiguity  in  the  content,  are: 

•  Badly  formatted  text  not  adhering  to  the  agreed  standard,  leading  to  parsing  errors. 
In  this  case,  a  human  operator  may  well  be  able  to  extract  information  from  the 
message  that  a  machine  cannot,  so  the  presence  of  this  sort  of  error  needs  to  be 
represented  in  some  way. 

•  Poorly  or  incorrectly  identified  incidents  or  units,  leading  to  incorrect  assignment 
of  reports  to  an  incident  or  unit.  While  this  may  be  a  difficult  situation  to  recognise, 
it  may  be  useful  to  represent  the  closest  known  match  for  a  unique  id. 

•  Missing,  erroneous  (e.g.  latitude  and  longitude  values  transposed),  or  incomplete 
(e.g.  MGRS  coordinates  not  specified  to  sufficient  precision)  location  information. 
In  some  cases  an  intelligent  system  may  try  to  correct  for  these  errors,  in  which 
case  this  provides  another  potential  source  of  unreliability  that  needs  to  be 
represented  in  some  way. 

•  Missing  or  ambiguous  date/ time  specification.  In  this  case  a  reasonable  default 
may  be  to  use  the  DTG  of  the  Mercury  header,  but  this  is  a  source  of  potential 
unreliability  that  may  need  to  be  represented  in  some  way. 

•  Inconsistent  or  unknown  named  entities  within  the  message  body.  This  can  lead  to 
poor  association  of  messages  with  events,  situations,  units,  etc.  While  this  may  be  a 
difficult  situation  to  recognise,  it  may  be  useful  to  represent  the  closest  known 
match  for  a  unique  named  entity. 

•  Inconsistent  information  within  a  message,  often  due  to  transcription  errors  (for 
example,  within  a  BBXX  message).  This  inconsistency  should  be  represented  in 
some  way. 

•  Corrections  to  MTF  message  are  routinely  broadcast,  often  leading  to  the  removal 
or  modification  of  previous  messages.  In  this  case,  some  form  of  representation 
that  a  correction  has  occurred  may  be  required. 

Network  latencies  may  introduce  a  significant  delay  between  the  reported  time  of  a  MTF 
message  and  the  time  that  it  is  available  to  the  system  as  a  'current'  update.  This 
introduces  a  potential  source  of  unreliability  that  may  need  to  be  represented  in  some  way. 
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6.3.12  Open-Source  RSS  feeds 

Rich  Site  Summary  (RSS)  and  the  similar  Atom  data  feeds  provide  structured  (XML) 
updates  of  information  that  is  time-stamped  and  accessible  to  web  applications.  The 
information  within  an  RSS  update  can  be  represented  as  free  text  or  as  (escaped)  XML.  In 
the  case  of  free  text,  similar  parsing  issues  and  potential  sources  of  uncertainty  to  MTF 
messages  arise,  the  particulars  of  which  depend  on  the  individual  RSS  feed.  In  the  case  of 
XML  content,  consistency  of  the  schema/ ontology  used  is  often  problematic,  even  with  a 
notional  'standard'  being  used,  as  different  sources  often  interpret  this  differently.  In  this 
case,  any  inconsistencies  between  RSS  feeds,  or  ambiguities  in  interpretation,  may  give  rise 
to  errors  during  the  data  transformation  stage  and  may  need  to  be  represented  or 
otherwise  accounted  for  in  some  way. 

GeoRSS  is  a  particularly  relevant  XML  schema  used  to  represent  geospatial  content  within 
RSS  and  atom  feeds  that  comes  in  two  variants:  Simple  and  GML.  Many  of  the  data  feeds 
of  use  to  SAKI  have  a  geospatial  component  represented  using  GeoRSS.  While  GeoRSS  is  a 
standard,  how  it  is  used  to  represent  geospatial  features  is  often  not  consistent.  For 
example,  a  region  affected  by  a  bushfire  could  be  represented  by  either  a  polygon  or  a 
closed  polyline,  or  set  of  polylines.  These  use  different  GeoRSS  data  structures,  so  this 
distinction  can  be  important  when  it  comes  to  interpreting  and/  or  representing  the  region. 

Latencies  of  updates  and  corrections  to  RSS  updates  are  possible  uncertainties  that  may 
also  need  to  be  represented. 

Finally,  there  are  potentially  many  RSS  feeds  reporting  updates  to  the  same  data,  or 
replicating  data,  or  publishing  variations  of  the  same  data  with  greater  or  lesser  reliability. 
Identifying  which  is  the  authoritative  data  for  particular  purposes,  and  which  is 
potentially  less  reliable,  may  be  important. 

6.3.1.3  Unstructured  documents 

SAKI  tries  to  use  information  in  unstructured  documents  intended  for  human 
consumption,  such  as  MS  Office  (Word,  PowerPoint,  Excel)  and  PDF  documents,  web 
pages,  etc.  Parsing  errors  similar  to  those  discussed  for  MTF  messages  can  occur,  but  in 
this  case  the  parsing  process  is  particularly  brittle  and  it  is  important  to  be  able  to  flag 
when  expected  information  may  be  missing  because  of  these  errors.  The  ability  to  show 
the  source  document  is  also  important  in  this  case. 

Unstructured  documents  are  generally  not  a  real-time  resource,  so  the  age  of  the 
information  contained  within  it  is  an  important  consideration,  and  the  ability  to  represent 
this  in  some  way  may  be  an  important  consideration  to  avoid  undue  emphasis  on  out-of- 
date  information. 

6.3.1.4  Structured  documents 

Information  can  also  be  sourced  from  structured  documents,  typically  in  XML  or  JSON 
formats,  which  are  often  generated  by  a  computer  program  or  from  databases  containing 
manually  entered  inputs.  As  such,  structured  documents  can  introduce  uncertainty 
through: 
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•  Data  entry  errors  giving  rise  to  inconsistent  data,  which  can  (sometimes)  be 
identified  due  to  inconsistency  with  the  document  schema,  and  in  this  case  it  may 
be  important  to  represent  this  inconsistency. 

•  Incomplete  metadata  associated  with  the  document.  Depending  on  the  nature  of 
the  metadata,  it  may  be  important  to  flag  that  it  is  incomplete.  This  can  also  give 
rise  to  errors  in  the  data  transformation  stage,  even  if  the  data  itself  is  good. 

•  Inconsistent  or  ambiguous  schema/ ontologies  between  documents,  which  may 
give  rise  to  errors  during  the  data  transformation  stage  that  may  need  to  be 
represented  or  otherwise  flagged. 

•  Out-of-date  information,  so  it  may  be  important  to  represent  the  currency  of  the 
information  in  some  way.  This  may  be  particularly  important  in  the  case  where  the 
document  represents  a  'snapshot'  of  a  database. 

6. 3. 1.5  Databases 

The  information  contained  within  databases  can  be  obtained  from  a  variety  of  sources: 

•  Manually  entered  by  a  human  operator 

•  Automatically  recorded  from  a  sensor  or  otherwise  automated  data  feeds  (e.g. 
track  data) 

•  Automatically  recorded  from  automated  data  processing 

•  Extracted  from  unstructured  documents,  including  from  'open-sources';  and 

•  Extracted  from  structured  documents. 

As  such,  many  of  the  issues  associated  with  the  data  sources  discussed  above  may  also 

apply  to  the  information  contained  within  databases: 

•  Data  entry  errors 

•  Incomplete  metadata  associated  with  data 

•  Inconsistent  or  ambiguous  schema/  ontologies  between  databases 

•  Out-of-date  information 

•  Conflicting  data  in  different  databases,  requiring  an  'authoritative'  source  to  be 
identified  and  represented  in  some  way 

•  Representing  and  indicating  corrections  made  to  database  data 

•  Representation  of  synthetic  vs  real  data  stored  in  databases 

•  'Noisy'  data  due  to  the  accumulation  of  various  errors  in  'big-data'  sets.  This  is 
particularly  relevant  when  parsing  errors  can  introduce  errors  in  the  information 
extracted  from  unstructured  text.  In  this  case,  the  reliability  of  the  data  needs  to  be 
represented  in  some  way. 
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6.3.2  Transformation 

In  SAKI,  the  primary  goal  is  to  extract  the  appropriate  information  from  the  available  data 
sources  and  integrate  it  to  provide  the  information  products  relevant  to  the  JOC  in  a 
relevant  manner.  This  requires  data  curation,  which  involves: 

•  Identifying  the  relevant  data  sources.  This  is  not  necessarily  a  'one-off'  process,  as 
linked  data  within  one  data  source,  particularly  when  dealing  with  open-source 
data,  could  specify  supplementary  data  sources  required  to  interpret  the  original 
data. 

•  Verifying  that  the  data  does  include  the  required  information  (for  example,  parsing 
errors  during  the  acquisition  stage  could  mean  that  some  data  is  missing) 

•  Cleaning  the  data  to  remove  invalid  subsets 

•  Transforming  the  data  into  the  required  formats 

•  Integrating  it  with  other  data  sources  of  interest.  This  includes:  associating  it  with 
other  data  related  to  an  'event'  of  interest;  handling  conflicting  data;  removing 
duplicates;  and  fusing  the  data  to  provide  the  required  information  products. 

Identifying,  verifying,  cleaning  and  transforming  data  are  the  processes  that  identify  and 
realise  some  of  the  errors  associated  with  the  original  data  sources,  and  allow  an 
evaluation  of  the  uncertainty/ reliability  of  the  information.  The  integration  process  treats 
some  of  this  uncertainty  (for  example  by  exploiting  the  authoritative  source  in  cases  of 
conflicting  data),  and  evaluates  how  this  uncertainty  affects  the  final  information  products. 

The  information  products  from  the  transformation  stage  can  be  varied,  and  include,  for 
example,  information  about  an  incident,  military  unit,  defence  capability,  defence 
establishment,  civilian  infrastructure,  or  a  weather,  geological,  or  fire  event.  The 
transformation  stage  can  introduce  its  own  errors  and  uncertainties  into  these  products: 

•  Association  of  information  used  to  provide  an  information  product  may  be 
incorrect,  particularly  when  dealing  with  unstructured  sources,  inconsistent 
schema/  ontologies  for  structured  data,  or  apparent  duplicates.  In  this  case,  it  may 
be  important  to  provide  some  representation  of  the  possible  alternative 
associations. 

•  Incomplete  information  required  to  provide  an  information  product  may  lead  to 
uncertainties  associated  with  that  product.  This  is  independent  of  the  reliability  of 
any  individual  data  source  used  to  make  up  that  product.  The  impact  on  the 
uncertainty  associated  with  the  product  is  highly  dependent  on  the  nature  of  the 
product,  but  it  may  be  important  to  represent  this  incompleteness  in  some  way. 

•  Aggregation  of  classified  information  can,  in  principle,  raise  the  classification  of  the 
aggregated  information  above  that  of  its  parts.  In  this  case,  the  information 
products  may  be  downgraded  (by  removal  of  information)  to  a  classification 
appropriate  for  the  operating  environment.  It  may  thus  be  important  to  provide 
some  representation  that  this  has  occurred  and  that  higher  classification  data  may 
potentially  be  available  on  another  system. 
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6.3.3  Visualisation 

SAKI  is  designed  to  allow  a  user  to  search  for  and/or  select  relevant  information,  and 
display  aggregated  information  about  the  results  found.  It  is  often  important  for  the  user 
to  be  able  to  validate  the  information  found,  and  manually  extract  related  information.  As 
such,  it  needs  to  display: 

•  Source  documents,  messages,  etc. 

•  Lists  and  summaries  of  search  results 

•  Tracks  and  geospatial  features 

•  Aggregated  multimedia  information  related  to  an  event,  military  unit,  defence 
capability,  defence  establishment,  or  civilian  infrastructure. 

•  Geospatial  context,  coordinates,  areas  of  effect  and  time  evolution  of  same. 

The  visualisation  artefacts/ ambiguities/ issues  that  can  be  introduced  with  these  displays 
are: 

•  How  can  you  visualise  content  that  didn't  parse  and  so  isn't  searchable,  but  may 
still  be  relevant  to  the  query? 

•  Search  results  are  often  'paged'  when  a  large  number  of  results  are  returned. 
Results  beyond  the  first  page  may  never  be  reviewed,  making  the  sorting 
algorithms  crucial. 

•  Dynamic  updates  to  search  results  can  cause  spontaneous  re-ordering  of  results, 
confusing  the  user. 

•  When  search  results  are  linked  to  a  geospatial  display,  depending  on  the  sorting 
criteria  this  can  give  a  misleading  impression  of  the  geospatial  distribution  of 
results  overall. 

•  A  geospatial  display  may  be  presented  at  an  inappropriate  scale,  excluding  or 
hiding  geospatial  context. 

•  A  geospatial  display  may  be  too  cluttered  for  a  user  to  extract  useful  information 
from  it. 

•  Geospatial  clustering  of  geospatially  adjacent  features  is  a  common  approach  to 
reducing  clutter  in  a  geospatial  display  at  different  scales.  Clustering  algorithms 
can  potentially  cluster  content  in  a  non-intuitive  way  (e.g.  it  may  cluster  purely  on 
spatial  separation  and  not  on  feature). 

•  Overlapping  map  layers  may  interfere  with  each  other,  producing  a  shading  or 
colourisation  effect  that  looks  nothing  like  the  source  layers,  hence  risking  incorrect 
interpretation.  Similarly,  overlapping  layers  may  mask  important  features. 

•  Overlapping  or  adjacent  map  symbols  may  actually  form  shapes  that  can  be 
mistaken  for  other  symbols. 
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6.4  Uncertainty  visualisation  for  JOC 

SAKI  deals  with  large  amounts  of  information  that  comes  from  many  sources  in  a  variety 
of  formats,  ranging  from  sensor  data  to  unstructured  documents.  One  big  challenge  in  the 
development  of  SAKI  is  how  to  effectively  fuse  and  display  numerous  instances  of 
uncertain  and  noisy  information  for  better  situation  awareness  and  decision  making.  The 
commander  must  make  decisions  under  time  pressure,  but  ignoring  the  fact  of  information 
uncertainty  could  lead  to  severe  consequences  in  the  military  domain.  SAKI  exposes  many 
issues  relevant  to  dealing  with  uncertainty  in  JOC,  which  include: 

•  It  is  necessary  to  develop  visualisation  methods  in  SAKI  for  depicting  multiple 
kinds  of  uncertainty.  As  shown  in  Table  1  (Page  3),  uncertainty  can  be  classified 
into  three  types  as  spatial,  temporal  and  attributes  of  data.  However,  these 
categories  are  often  interdependent,  and  the  category  boundaries  are  hard  to 
delineate  (MacEachren  et  al.,  2005).  This  raises  the  issue  of  how  to  visualise 
compounded  uncertainty  information.  For  example,  geo-references  for  a  particular 
bushfire  event  (see  Figure  28)  could  include  three  types  of  uncertainties:  spatial 
uncertainty  in  its  position  due  to  fuzzy  descriptors;  temporal  uncertainty  due  to 
low  reporting  rates;  and  attribute  uncertainty  in  its  status  due  to  unreliable  or 
conflicting  reports.  In  this  case,  different  visualisation  techniques  could  be  applied 
to  represent  each  aspect  of  uncertainty.  As  discussed  in  Sections  3  and  4,  temporal 
uncertainty  could  be  represented  by  opacity,  and  spatial  uncertainty  could  be 
represented  by  blurring.  Uncertainty  in  its  status  (e.g.  Out  of  Control/ Contained) 
could  be  depicted  by  using  degraded  icons.  However  the  effectiveness  of  this 
potential  approach  should  be  verified  by  empirical  studies  to  see  whether  the 
operator  maintains  situational  awareness  of  the  different  uncertainties  in  data. 


Figure  28:  Screenshot  of  SAKI' s  geospatial  visualisation  system 
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Most  work  cited  in  this  report  has  focused  on  numerical  uncertainties  such  as 
temporal  and  spatial  error,  with  only  limited  attention  given  to  more  abstract 
forms  of  uncertainty  such  as  consistency  and  credibility.  As  discussed  in  the 
preceding  sections,  SAKI  also  has  to  represent  these  more  abstract  concepts.  For 
these  cases  it  may  (or  may  not)  be  sufficient  to  provide  a  visual  indication  to  the 
user  that  such  uncertainty  exists  through  highlighting  and/  or  colour  mapping. 

A  related  issue  is  how  to  represent  uncertainty  in  those  cases  where  the  SAKI 
system  has  been  unable  to  interpret,  for  various  reasons,  the  information  available 
to  it.  In  some  cases  SAKI  may  need  to  advise  the  user  that  such  information  exists, 
represent  the  nature  of  the  incompleteness  and  provide  an  appropriate 
visualisation.  For  example,  badly  formatted  geospatial  coordinates  would  mean 
that  there  is  no  geospatial  representation  of  the  information  on  the  SAKI  geo 
display.  This  could  potentially  be  represented  using  an  interactive  notification 
which  provides  elaboration  on  the  information  that  is  available,  and  what  the 
problem  with  it  is,  when  the  user's  mouse  hovers  over  or  clicks  it. 

The  literature  has  shown  that  experts  and  novices  cope  differently  with  less  reliable 
information  in  decision  making,  and  may  require  different  visualisations  (Kobus  et 
al.,  2001,  John  et  al.,  2000,  Omodei  et  al.,  2005).  It  is  unlikely  that  all  users  of  SAKI 
are  subject  matter  experts  in  all  of  the  different  data  sets,  and  further,  in  every  task 
they  are  required  to  do.  The  users'  level  of  experience  needs  to  be  considered  in  the 
selection  of  uncertainty  visualisation  approaches  and  interface  design,  perhaps  by 
incorporating  user  models  that  tailor  the  display  to  suit  their  level  of  experience 
and  role. 

User  preferences  for  the  representation  of  uncertainty  have  been  shown  to  be 
correlated  with  the  performance  of  decision  making  (Gerharz  and  Pebesma,  2009). 
It  is  important  to  provide  a  user  interface  which  allows  users  to  intuitively  access 
and  modify  the  uncertainty  representation  approach  used,  according  to  operational 
roles  or  situation  changes: 

o  The  user  interface  should  allow  management  of  individual  overlays, 
cropping  of  unnecessary  data  sets  and,  adjustment  of  the  interface  design  or 
visualisation  parameters  to  focus  on  important  uncertainty  features 

o  It  would  be  desirable  to  provide  multiple  ways  of  displaying  uncertainty 
attributes  to  support  different  user  preferences  and  manage  cognitive  loads 
for  different  roles  and  tasks.  Flowever,  as  discussed  in  §4.2.2,  users  may  not 
always  prefer  the  most  effective  representation,  so  some  constraints  would 
need  to  be  applied  to  user  configuration,  perhaps  based  on  user  and  task 
models. 

o  Similarly,  users  may  not  be  the  best  judge  of  their  situation  awareness 
requirements  and  when  they  have  enabled  display  of  too  much  information 
(Omodei  et  al.,  2005).  It  may  be  useful  to  incorporate  into  the  display  one  or 
more  of: 

■  metrics  that  allow  the  user  to  monitor  their  own  performance 
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■  automated  monitors  that  alert  them  when  they  may  have  added  too 
much  to  the  display 

■  constraints  that  restrict  how  many  dimensions  of  the  data  they  can 
display  at  any  given  time. 

•  An  avatar  could  be  used  to  provide  high-level  narratives  of  a  situation  and  convey 
different  levels  of  reliability  or  uncertainty  to  users.  This  may  also  be  appropriate, 
for  example,  when  summarising  the  status  of  an  evolving  event.  Other 
visualisation  modalities  may  also  be  appropriate  for  conferring  different  levels  of 
confidence  in  uncertain  information. 

•  Multimodal  interfaces  could  be  incorporated  in  SAKI  to  display  a  large  amount  of 
data  by  reducing  the  dimensionality  of  the  data  that  needs  to  be  presented  visually. 
In  particular,  multimodal  channels  could  enhance  the  users'  ability  to  process 
information  simultaneously  and  direct  attention  to  critical  information.  For 
example,  an  audible  alarm  could  alert  users  when  a  bushfire  changes  status  to  'Out 
of  Control',  or  aural  channels  could  be  used  to  represent  otherwise  overlapping 
visual  layers  (such  as  weather  and  fire-affected  regions)  which  could  cause 
interference  between  shading  and  colouring  effects. 

•  An  evaluation  framework  is  needed  to  investigate  the  impact  of  visualisation 
approaches  in  general,  and  uncertainty  visualisation  in  particular,  on  the  users' 
situational  awareness.  For  the  JOC  case,  factors  that  need  to  be  taken  into  account 
include: 

o  Numerous  and  repetitive  user  tests  may  be  required  to  iteratively  evaluate 
and  refine  a  visualisation  approach,  because  of  the  lack  of  relevant 
empirical  studies  or  comprehensive  guidelines. 

o  It  is  crucial  to  recruit  participants  who  are  experienced  with  the  tasks 
undertaken,  and  who  can  provide  feedback  on  practical  and  effective 
system  designs.  This  also  might  be  useful  to  study  the  effect  of  experience 
on  decision  making  with  uncertainty.  The  pool  of  experienced  users  for  the 
JOC  case  is  limited,  and  they  will  have  limited  time  available  to  participate 
in  system  evaluation. 

o  Further  to  this,  the  information  requirements  for  the  JOC  case  are  diverse 
and  highly  dynamic,  so  opportunities  to  gain  experience  with  a  particular 
role  (and  its  information  requirements)  are  limited.  However,  this  also 
means  that  the  inclusion  of  uncertainty  visualisation  may  provide 
significant  improvements  to  the  user's  decision  making  performance. 

o  Better  understanding  is  needed  of  the  cognitive  processes  for  interpreting 
information  with  uncertainty  so  that  effective  mechanisms  can  be  applied  to 
meet  the  diverse  and  dynamic  requirements  for  JOC. 

•  Reducing  the  sources  of  uncertainty  for  SAKI  could  mitigate  the  requirement  to 
represent  them.  Approaches  to  this  could  include: 
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o  Development  and/or  application  of  more  advanced  parsing  algorithms 
using  recent  advances  in  machine  learning,  for  a  larger  variety  of 
documents,  to  deal  with  ambiguities  in  language  and  interpretation  of 
context. 

o  Establishment  and  wider  community  uptake  of  consistent  standards  for 
schema/ ontologies  to  avoid  inconsistent  or  incorrect  interpretation  of 
information  sources,  and  to  support  meaningful  association  of  information, 
including  standardisation  of  geospatial  formats  (e.g.  polygon 
representations).  The  wider  community  awareness  of  'big  data'  and  data 
science  approaches  shows  promise  in  this  regard. 
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7.  Conclusion 


Military  commanders  are  typically  required  to  make  critical  decisions  and  develop  plans, 
using  large  and  complex  collections  of  data,  in  a  limited  time  frame.  They  must  do  this 
without  precise  knowledge  about  the  operating  environment  or  the  intent,  capabilities  or 
location  of  the  adversary.  It  is  important  that  commanders  understand  the  associated 
uncertainties  so  that  they  can  understand  and  mitigate  the  operational  risks  involved  in 
this  inherently  risky  enterprise.  Therefore  it  is  important  for  decision  support  tools  to 
make  users  aware  of  the  uncertainties  in  the  information  being  displayed  in  an 
appropriate  and  timely  manner,  while  avoiding  information  overload.  However,  most 
visualisation  techniques  have  been  designed  around  the  assumption  that  the  data  being 
visualised  is  free  from  uncertainty. 

This  report  introduced  the  main  conceptualisations  and  approaches  for  representing 
uncertainty  suggested  in  the  literature.  There  are  eleven  categories  of  uncertainty  that  have 
been  identified  in  the  literature  that  relate  to  different  information  requirements.  These 
uncertainties  can  be  introduced  during  the  data  acquisition,  data  transformation,  or  data 
visualisation  processes.  How  uncertainty  needs  to  be  represented  depends  on  the  nature 
and  source  of  the  uncertainty,  and  a  number  of  visualisation  techniques  have  been 
presented.  However,  there  is  an  enduring  need  to  ground  these  techniques  in  perceptual 
and  cognitive  theory,  and  more  systematic  empirical  studies  into  the  effectiveness  of  these 
techniques  are  needed  before  results  can  be  generalised  to  other  application  contexts. 

Empirical  studies  carried  out  on  the  effect  of  uncertainty  on  decision  making  have  found 
marked  differences  in  performance  between  experienced  and  inexperienced  users.  The 
decision  times  for  experienced  users  were  not  affected  by  uncertainty,  while  the  decision 
times  for  inexperienced  users  were  significantly  longer.  Interestingly,  visualisation  of 
uncertainty  has  been  found  to  improve  decision  making  performance  for  relatively  easy 
tasks,  but  not  when  dealing  with  more  difficult  tasks.  Perhaps  in  these  cases  other 
considerations  dominate.  Studies  have  also  shown  that  user  experience  is  particularly 
important  in  the  selection  of  participants  for  empirical  studies  into  uncertainty 
representation  techniques.  Clearly,  the  skills  and  experience  of  participants  in  these 
empirical  studies  need  to  be  representative  of  the  skills  and  experience  of  the  target  user 
group,  which  could  be  problematic  for  applications  targeted  at  operational  commanders, 
who  have  a  unique  skill  set. 

Several  empirical  studies  have  also  looked  at  the  effectiveness  of  different  uncertainty 
visualisation  approaches  on  decision  making  in  different  contexts.  Dynamic  uncertainty 
representation  techniques  (e.g.  animation)  were  generally  found  to  be  less  effective  than 
static  techniques  (e.g.  glyphs)  in  decision  making  tasks.  Somewhat  surprisingly,  the 
addition  of  textual  annotations  of  confidence  to  a  glyph  was  found  to  give  poorer 
performance  than  the  glyph  alone  some  tasks,  perhaps  indicating  the  presence  of 
information  overload.  In  other  studies,  different  uncertainty  representations  were  found  to 
suit  different  user  requirements  in  the  same  application,  suggesting  that  some  degree  of 
tailoring  may  be  required  to  meet  multiple  user  roles.  This  should  be  treated  with  caution 
however,  as  other  studies  have  also  shown  that  user  satisfaction  with  information 
products  does  not  necessarily  coincide  with  improved  situational  awareness  in  general. 
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and  so  user  preferences  for  different  uncertainty  representation  techniques  do  not 
necessarily  coincide  with  best  decision  making  performance.  System  metrics  and 
governance  frameworks  may  be  needed  to  help  manage  this  complexity. 

Other  approaches  to  uncertainty  visualisation  could  use  the  method  of  representation 
itself  to  attribute  confidence  in  the  information  presented.  For  example,  users  have  greater 
confidence  in  information  when  it  is  presented  textually  than  when  it  is  presented  by  a 
virtual  human.  Thus,  information  with  low  uncertainty  could  be  presented  to  the  user  as 
text,  while  information  with  high  uncertainty  could  be  presented  to  the  user  by  an  avatar. 
Further  studies  could  also  explore  how  users  attribute  confidence  to  information 
presented  using  other  visualisation  modalities. 

General  guidelines  for  visualisation  can  be  applied  to  help  formulate  an  uncertainty 
visualisation  approach  suitable  for  a  particular  context,  once  the  relevant  components  of 
uncertainty,  their  relationships  to  the  data,  and  the  desired  decision  making  outcomes  are 
understood.  Flowever,  there  are  limited  guidelines  available  for  representing  multiple 
types  of  uncertainty  in  decision  making  applications  in  particular.  More  study  is  needed  in 
how  to  depict  multiple  forms  of  uncertainty  in  the  same  display,  and  how  providing 
multiple  types  of  uncertainties  affects  the  users'  understanding  of  the  information.  User 
centred  design,  iterative  approaches  to  development,  and  methods  for  assessing  relevant 
performance  metrics,  are  considered  crucial  given  the  sensitivity  of  decision  making 
outcomes  to  context,  user  experience,  and  user  roles. 

Some  of  these  considerations  have  been  discussed  in  the  context  of  the  situational 
awareness  for  the  Joint  Operations  Command,  although  empirical  studies  and  further 
conceptual  refinement  are  needed  to  target  their  particular  user  requirements.  Significant 
barriers  to  uncertainty  visualisation  research  in  this  context  are  the  availability  of 
experienced  users  for  empirical  evaluation,  and  the  dynamic  nature  of  their  visualisation 
requirements  across  a  broad  range  of  information  types.  The  dynamic  nature  of  their 
requirements  means  that  many  of  these  users  may  be  relatively  inexperienced  with  a 
particular  task  or  display,  which  may  make  it  easier  to  find  experiment  participants 
representative  of  this  target  group.  It  also  indicates  that  visualisation  of  the  associated 
uncertainties  may  provide  significant  benefits  to  decision  making  performance  in  this 
context. 

Finally,  how  decision  making  with  uncertainty  translates  to  operational  risk  is  an  area  that 
requires  further  study.  In  the  studies  presented  in  the  literature  the  performance  of  the 
decision  maker  was  evaluated  using  metrics  such  as  response  time  and  the  user's 
confidence  in  the  outcome.  Whether  the  representation  of  uncertainty  translates  to  a  better 
appreciation  of  operational  risk,  and  more  effective  mitigation  strategies,  has  not  been 
considered  in  this  work.  Studies  that  model  and  evaluate  operational  risk  could  help 
address  this  question. 
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